By default, records with an empty-string value in a join field are
paired just like any other value, so two records both missing an ID
get matched with each other -- rarely the intended behavior.
--ignore-empty treats an empty-string join-field value as if the
field were absent, on both the left and right files, so such records
fall through to unpaired handling (--np/--ul/--ur) instead of
cross-joining on the empty string.
Wires the check through both the default half-streaming join and the
-s/--sorted-input doubly-streaming join, which track left-file
buckets independently and needed the same empty-aware key-presence
check in JoinBucketKeeper.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The reader's three-pass pipeline (backslash-join -> fold "+"-continuations
onto raw lines -> split each folded line on ": ") broke on fields written
as bare "Name:" (colon, no trailing space, no value) whose value is
supplied entirely by following "+" lines -- the idiomatic recutils pattern
for e.g. a "%doc:" field. Folding the continuation onto the raw line
destroyed the ": " substring needed for the later split, so parsing such a
field crashed with a "missing field separator" error, even on well-formed
input.
Found by testing against GNU recutils' own tutorial example (books.rec,
from the manual's "A Little Example" page), now added as a fixture.
Restructured parsing to split each line into a key/value field before
folding continuations, so folding happens on the value rather than the
raw line; when the preceding value is empty, the continuation becomes the
value outright instead of gaining a spurious leading newline.
Adds regression coverage (test/cases/io-recutils/0006, 0007) and a docs
section covering the empty-value + continuation pattern.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Implements the plan in plans/recutils.md, addressing #378. Adds a
hand-rolled reader/writer pair (pkg/input/record_reader_rec.go,
pkg/output/record_writer_rec.go) supporting recutils' blank-line-separated
"Key: value" records, "+"-continuation and backslash-newline continuation,
and generic comment handling, with recutils' %rec-descriptor schema layer
left unspecialized since Miller has no schema-enforcement concept.
Registers the format under the "recutils" name (--irecutils/--orecutils/
--recutils flags, matching the --idcf/--odcf/--dcf pattern) in the
reader/writer factories and pkg/cli/separators.go, adds docs and a sample
data file, and adds unit tests plus test/cases/io-recutils regression
cases covering round-tripping, continuation lines, comments, and the
hard-error behavior on malformed input.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Add named profile sections to .mlrrc, selected via --profile / -P (#358)
.mlrrc files may now contain INI-style named sections ("profiles"):
# Global settings, applied always:
icsv
[j]
ojson
jvstack
[tsvout]
otsv
Lines before any section header are global settings, applied always, so
existing .mlrrc files behave exactly as before. The new main flag
--profile {name} (alias -P {name}) applies the settings from the [name]
section after the global settings; without it, sections are ignored
entirely (their lines aren't even parsed).
It's a fatal error if a requested profile has no matching section in any
.mlrrc file processed, if no .mlrrc file was found at all, or if
--profile is combined with --norc or MLRRC=__none__.
Also fixes the regression-tester to restore per-case environment
variables to their prior values after each case, rather than setting
them to the empty string -- needed so cases pointing MLRRC at a test
file don't clobber the suite-wide MLRRC=__none__ guard.
Includes unit tests, regression-test cases, and doc/man-page updates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Reject --profile / -P within .mlrrc files (#358)
Per maintainer feedback on the PR: profiles are selected on the mlr
command line, not from within a .mlrrc file, so --profile / -P inside a
.mlrrc file is now a parse error -- the same way --prepipe is rejected
there -- rather than being silently ignored.
Adds a unit test, a should-fail regression case, and a docs bullet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* stats1: add rank accumulator (#383)
Adds `mlr stats1 -a rank` for standard competition ranking (1,2,2,4,...)
on pre-sorted data, most useful with -s for a rank on every record.
* stats1: make rank order-independent by default, add --rank-sorted opt-in fast path
The rank accumulator previously only compared each value to the immediately
preceding record, silently giving wrong ranks for non-adjacent duplicates
(e.g. unsorted input, or interleaved -g groups). Default to correctly
computing standard competition rank from all values seen so far
(order-independent, buffers values, same approach as percentile
accumulators). Add --rank-sorted for callers who can promise sorted input
and want the previous O(1)-space streaming behavior instead.
* Revert "stats1: make rank order-independent by default, add --rank-sorted opt-in fast path"
This reverts commit aa45a591fe.
* Revert "stats1: add rank accumulator (#383)"
This reverts commit 96deed048a.
* Add mlr rank verb (#383)
Reverts the earlier stats1 -a rank / --rank-sorted approach: stats1 is a
reduce verb (many records -> one summary record per group) and rank is a
per-record annotator, so it never fit cleanly there -- it needed stats1's
-s iterative-stats escape hatch just to be useful, plus a bolted-on
sorted/unsorted split.
mlr rank is a dedicated verb modeled on mlr fraction: -f fields to rank,
-g optional group-by, output field <f>_rank. By default it's a two-pass
algorithm (buffers input, like fraction does) giving standard competition
rank (1,2,2,4,...) that's correct regardless of input order. --sorted
opts into a single-pass, O(1)-space streaming alternative for callers who
can promise pre-sorted input (e.g. via 'mlr sort' first).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate docs/man pages after merging main (sparkline verb)
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The join verb's left-file ingest silently swallowed read errors: a
nonexistent or malformed left file produced empty (or partial) output
with exit code 0.
Two defects, one per ingest path:
* Unsorted (half-streaming) path, ingestLeftFile in
pkg/transformers/join.go: errors from the record-reader's error
channel were dropped ("TODO: propagate error to caller"), as were
record-reader construction errors. All left-file read failures were
silent.
* Sorted (-s) path, readRecord in
pkg/transformers/utils/join_bucket_keeper.go: the error channel was
consulted, but the record-reader goroutine sends the error and the
end-of-stream marker on separate channels, so the select could see
end-of-stream first and drop the error. A nonexistent left file with
-s exited 0 about half the time (racy).
Both paths now print "mlr: <error>" to stderr and exit 1, matching how
read errors on regular (right) input files are reported. On receipt of
the end-of-stream marker, both consumers now do a final non-blocking
drain of the error channel, which is deterministic since the reader
sends any error before the marker.
Discovered while verifying #377.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
With multiple regexes, 'mlr reorder -r' previously emitted all matched
fields in record order, ignoring the order the regexes were given. Now
matched fields are grouped by regex-list order (first regex's matches
first, then the second's, etc.); within each group, fields keep their
record order. A field matching multiple regexes is claimed by the first
one. This makes 'reorder -r' consistent with 'cut -orf' and satisfies
the original request in #1325: -r '^YYY,^XXX' puts YYY-prefixed fields
first, then XXX-prefixed fields, then the rest. Applies to -e, -b, and
-a modes as well.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The --lp/--rp prefixes and -j output-field rename were applied only to
paired records, leaving unpaired records emitted via --ul/--ur with their
original (left/right) field names. This broke downstream operations like
unsparsify that expected consistent column names across paired and
unpaired output.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The implicit-header readers for TSV and CSV-lite were missing the
`for` loop in their `nh < nd` branch, so when a data row had more
fields than the auto-generated header, only one extra field was
captured and the rest were dropped. Restore the loop, matching the
explicit-header readers and the pprint reader.
When using --fr (regex field selector) with stats1 -a null_count, void
(empty) field values were unconditionally skipped in
ingestWithValueFieldRegexes, causing null_count to always report 0.
The non-regex path (ingestWithoutValueFieldRegexes) already had a special
case that allows void values through for null_count accumulators. This
commit adds the same exception to the regex path.
Fixes#1639
Co-authored-by: cobyfrombrooklyn-bot <cobyfrombrooklyn-bot@users.noreply.github.com>
* reader
* record writer
* pkg/output/record_writer_dcf.go
* test/input/test.dcf
* test/cases/io-dcf/
* pkg/cli/option_parse.go
* make fmt
* make dev
* docs and `make dev`
Add a surv verb to estimate a survival curve using Kaplan-Meier. It
requires duration and status (event or censored) columns, and outputs
each distinct duration and corresponding probability of survival.
* DSL stats functions [WIP]
* refactor
* move percentile computation to bifs module; iterate
* mode and antimode
* percentile iterate
* percentile sketching
* neaten
* unit-test iterate
* unify old & new min & max functions
* unit-test cases
* code-dedupe between mode and antimode
* make mode/antimode ties deterministic via first-found-wins rule
* online help strings for new stats DSL functions
* artifacts from `make dev`
* help info on how min/max now recurse into collections
* artifacts from `make dev`
* typofix
* DSL functions for 64-bit nano-epoch timestamps
* strfntime
* nsec2gmt; move sec/nsec pairs adjacent to one another
* update on-line help
* artifacts from `make dev`
* unit-test files
* latin1_to_utf8 and utf8_to_latin1 DSL functions
* doc-build artifacts for previous commit
* Test cases for latin1_to_utf8 and utf8_to_latin1
* extend on-line help
* latin1_to_utf8 and utf8_to_latin1 verbs
* unit-test cases for verbs
* Keep with kebab-case naming convention for verbs
* webdocs
* Add natural sort order as an option for the sort verb
* Add natural sort order as an option for the sort DSL function
* doc-build artifacts for on-line help
* webdocs
* codespell fix
* unit-test files for sort verb
* unit-test files for sort DSL function
* todo
* Neaten existing DSL sketch
* rebase on #893, and sketch
* code-complete
* build artifacts for previous commit
* replace - with _ in shift and slwin
