When a transformer fails mid-stream (e.g. join -s with a malformed left
file), the error could be lost, yielding exit 0 with no stderr message:
- runSingleTransformerBatch forwarded the end-of-stream marker downstream
before runSingleTransformer sent the error to
dataProcessingErrorChannel, so the record-writer could finish and signal
done-writing while the error was still unsent.
- Even with the error buffered, stream.Stream's select loop chooses among
simultaneously-ready channels at random, and exiting on the done-writing
signal dropped the buffered error.
Send the error before forwarding the end-of-stream marker (so it is
always buffered before the writer can finish), and drain the error
channels after the select loop exits.
Observed as a one-off Windows CI failure of
test/cases/verb-join/left-file-malformed-sorted, where the same case
passed on automatic rerun within the same job. Reproduced locally by
widening the deschedule window with a sleep between the end-of-stream
forward and the error send: 5/5 runs exited 0 with empty stderr; with
this fix, 200/200 runs exit 1 with the expected message even with
adversarial delays injected on both sides of the end-of-stream forward.
Follow-up to the os.Exit-removal refactor (plans/exit.md, #2204/#2205).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
By default, records with an empty-string value in a join field are
paired just like any other value, so two records both missing an ID
get matched with each other -- rarely the intended behavior.
--ignore-empty treats an empty-string join-field value as if the
field were absent, on both the left and right files, so such records
fall through to unpaired handling (--np/--ul/--ur) instead of
cross-joining on the empty string.
Wires the check through both the default half-streaming join and the
-s/--sorted-input doubly-streaming join, which track left-file
buckets independently and needed the same empty-aware key-presence
check in JoinBucketKeeper.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Phase-5 wrap-up for plans/exit.md. The DSL-runtime cluster (pkg/dsl/cst:
hofs.go, udf.go, evaluable.go) is deferred rather than converted: 58
regression cases pin the current loud-failure behavior (exact stderr plus
exit 1), and the planned error-Mlrval mechanism would make HOF/UDF misuse
fail silently as bare '(error)' with exit 0 -- a debugging-UX regression.
The alternative (typed panic recovered at the Execute* boundaries) preserves
behavior but introduces panic/recover to a codebase that has none. The
decision belongs with the issue-440 strict-mode design, where the
fatal-vs-data error taxonomy gets decided anyway; plans/exit.md now records
the rationale, the phase-by-phase status (#2198, #2202, #2204, #2205), and
the final intentional keep-list.
Also converted here, since they're squarely in prior phases' patterns rather
than the deferred expression-depth cluster:
- RootNode.ProcessEndOfStream (a phase-3 leftover on the put/filter
Transform path) returns an error instead of printing-and-exiting on
end-of-stream close failures; first error returned, any others printed at
the site.
- The three CompileMillerRegexOrDie calls in option_parse.go (inside
error-returning parser closures since phase 2) use CompileMillerRegex and
return the error; 'mlr --ifs-regex (' output and exit code are unchanged.
All 4779 regression cases pass; make lint 0 issues.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Phase 4 of plans/exit.md: the aux/terminal sub-entrypoints. pkg/auxents and
pkg/terminals are now os.Exit-free; even the dispatchers return exit codes
instead of exiting.
- auxents.Dispatch returns (handled bool, exitCode int); entrypoint.Main --
the one sanctioned exit point -- exits with the code.
- terminals.Dispatch returns an exit code; the climain caller wraps it in a
lib.ExitRequest, which also retires the 'terminal did not exit the process'
panic.
- Usage functions (hex, unhex, lecat, termcvt, repl, script, regtest) no
longer take an exitCode parameter and exit; call sites print usage and
return the code explicitly.
- Inner I/O helpers (hexDumpFile, unhexFile, lecatFile, termcvtFile) return
errors; each sub-main prints 'mlr {verb}: ...' as before and returns 1.
lecatFile formerly looped forever on a non-EOF read error; it now returns
the error.
- The regtest directory walk (Execute / executeSinglePath /
executeSingleDirectory / hasCaseSubdirectories) plumbs errors up to
RegTestMain; regression_test.go updated for the new Execute signature.
Stderr messages and exit codes are unchanged across mlr aux-list, hex, unhex,
lecat, termcvt, help, version, repl, script, and regtest surfaces (checked by
hand); all 4779 regression cases pass; make lint is clean; docs rebuild with
zero churn.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Phase 3 of plans/exit.md: the streaming-interface change, the load-bearing
piece for #341 (DSL exit statement) and #440 (strict mode).
- RecordTransformer.Transform and RecordTransformerFunc now return error.
All 69 Transform implementations and their dispatch helpers updated
(mechanical rewrite, compiler- and errcheck-verified).
- runSingleTransformerBatch, on a Transform error, forwards any output
produced before the failure plus an end-of-stream marker downstream, so
the rest of the chain and the record-writer drain and finish cleanly;
runSingleTransformer then surfaces the error to stream.Stream's select
loop (non-blocking send; first error wins) and signals upstream-done so
the record-reader stops. This is exactly the flush-then-exit sequencing a
future DSL 'exit N' needs.
- dataProcessingErrorChannel and FileOutputHandler.recordErroredChannel are
now chan error instead of chan bool; ChannelWriter still prints write-
error details at the site and sends the 'exiting due to data error'
sentinel, preserving the exact stderr shape pinned by regression cases.
- Mid-stream os.Exit sites converted to returned errors: put/filter DSL
begin/main/end-block errors and the non-boolean filter-expression case,
tee write/close failures, split write/open/close failures, join left-file
ingest failures (both half-streaming and sorted paths, with full error
plumbing through JoinBucketKeeper), histogram/stats2 ingest errors, surv
fit errors, and step stepper allocation (tStepperAllocator now returns
(tStepper, error); bad EWMA coefficients propagate; negative slwin
parameters are reported by the CLI parser via the existing
bad-stepper-name pattern).
- The two genuinely internal join-bucket-keeper states now use
lib.InternalCodingErrorWithMessageIf instead of hand-rolled print+exit.
- pkg/transformers is now os.Exit-free.
Behavior notes: the non-boolean filter message gains the standard 'mlr: '
prefix and a newline (it previously printed with neither); tee errors now
include the underlying cause. All 4779 regression cases pass unchanged;
mlr head early-out latency is unaffected (0.02s over 50M records).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The reader's three-pass pipeline (backslash-join -> fold "+"-continuations
onto raw lines -> split each folded line on ": ") broke on fields written
as bare "Name:" (colon, no trailing space, no value) whose value is
supplied entirely by following "+" lines -- the idiomatic recutils pattern
for e.g. a "%doc:" field. Folding the continuation onto the raw line
destroyed the ": " substring needed for the later split, so parsing such a
field crashed with a "missing field separator" error, even on well-formed
input.
Found by testing against GNU recutils' own tutorial example (books.rec,
from the manual's "A Little Example" page), now added as a fixture.
Restructured parsing to split each line into a key/value field before
folding continuations, so folding happens on the value rather than the
raw line; when the preceding value is empty, the continuation becomes the
value outright instead of gaining a spurious leading newline.
Adds regression coverage (test/cases/io-recutils/0006, 0007) and a docs
section covering the empty-value + continuation pattern.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Remove os.Exit callsites below the entrypoint: phase 2 (plans/exit.md)
Phase 2 of plans/exit.md: the FlagParser signature change.
- FlagParser gains an error return; all 259 inline parser closures in
option_parse.go and NoOpParse1 updated (mechanical rewrite, compiler-checked).
- FlagTable.Parse returns (bool, error); its nine callers (climain passes one
and two, .mlrrc line handling, verb-local flag parsing in tee/join/put/
filter/split, and the repl/script terminals) propagate the error instead of
letting parsers kill the process.
- cli.CheckArgCount returns its missing-argument message as an error rather
than printing and exiting; new cli.FlagErrorf (counterpart of VerbErrorf)
carries user-facing main-flag messages.
- The 14 print-and-exit sites inside parser closures become returned errors;
--list-color-codes/--list-color-names return ExitRequest{0} after printing.
- The repl and script terminals translate ExitRequest into their int return
codes, so 'mlr repl --list-color-codes' still exits 0.
- pkg/cli is now os.Exit-free; stale comments about exiting parsers updated
(the completion introspection accessors remain, since parsers mutate the
options struct).
Stderr messages and exit codes are byte-identical, including the two-line
missing-argument and --seed messages and the trailing-period forms pinned by
test/cases/cli-mfrom/0003.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Merge main (recutils #2201); convert its three new parser closures
Same mechanical rewrite as the rest of phase 2: error return signature plus
final 'return nil' on --irecutils/--orecutils/--recutils.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Implements the plan in plans/recutils.md, addressing #378. Adds a
hand-rolled reader/writer pair (pkg/input/record_reader_rec.go,
pkg/output/record_writer_rec.go) supporting recutils' blank-line-separated
"Key: value" records, "+"-continuation and backslash-newline continuation,
and generic comment handling, with recutils' %rec-descriptor schema layer
left unspecialized since Miller has no schema-enforcement concept.
Registers the format under the "recutils" name (--irecutils/--orecutils/
--recutils flags, matching the --idcf/--odcf/--dcf pattern) in the
reader/writer factories and pkg/cli/separators.go, adds docs and a sample
data file, and adds unit tests plus test/cases/io-recutils regression
cases covering round-tripping, continuation lines, comments, and the
hard-error behavior on malformed input.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Rebuild docs: help-catalog index count drifted on main (666 -> 667)
Pre-existing drift from a recent catalog addition; surfaced by make dev.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove os.Exit callsites below the entrypoint: phase 1 (plans/exit.md)
Phase 1 of plans/exit.md: the mechanical swaps, plus the single-exit-point
scaffolding.
- New sentinel lib.ExitRequest{Code} (io.EOF-style control flow): returned by
code paths that have already printed what they wanted (--version, -h,
'put --explain' on a valid expression, 'put -x') instead of exiting mid-stack.
- pkg/entrypoint: new exitOnError() maps errors to exit codes in one place:
ErrHelpRequested -> 0, ErrUsagePrinted -> 1, ExitRequest -> its code,
anything else printed (JSON if --errors-json) -> 1.
- pkg/climain: version/help/usage/validation exits become returned errors;
the ErrHelpRequested/ErrUsagePrinted -> exit translation moves up to the
entrypoint; loadMlrrcOrDie becomes error-returning loadMlrrcFiles.
- Transformer ParseCLI/constructor exits -> returned errors (cut,
having_fields, nest, reorder, merge_fields, reshape, rename, split, tee,
subs, put_or_filter). merge_fields' bad-regex message formerly mis-reported
itself as coming from the cut verb; tee's unrecognized-option exit was
formerly silent; split/tee constructor failures formerly exited without
printing the constructor's error.
- YAML/JSON record-writer marshal errors -> returned through Write, matching
the CSV writer's error path.
- lib.WriteTempFileOrDie -> WriteTempFile (string, error); its sole caller
(regtest diff helper) degrades gracefully.
Stderr messages and exit codes are byte-identical for all regression-covered
paths (4779 cases pass); runtime Transform-path exits are phase 3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add lib.NewExitZeroRequest() constructor for exit-0 sentinel returns
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Treat YAML like JSON for auto-flatten/auto-unflatten (#2195)
Auto-flatten/unflatten decisions only checked for "json", so nested
data written to YAML was needlessly flattened and non-JSON data
converted to YAML was never unflattened into arrays/maps. Both formats
support native nesting, so both should be treated the same way.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Clarify why DCF is excluded from auto-flatten separately from JSON/YAML
DCF isn't a nestable format like JSON/YAML -- it just has its own
hardcoded comma-list serialization for a fixed set of field names
that generic key-spreading flatten would clobber. Split the comment
so that distinction isn't lost.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Refer to JSON/YAML throughout the flatten-unflatten doc
The summary paragraph calling out YAML as an aside did not read
naturally next to prose that otherwise only said JSON. Fold YAML
into the headings and body text directly instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
docs/src/mlr.1 is a leftover copy of the man page that no build target
generates or consumes: the maintained man page is man/mlr.1, produced by
man/Makefile (mkman.rb), and nothing in docs/ or any Makefile references
the docs/src copy. It has drifted stale, which multiple contributors have
independently noticed while regenerating docs.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
NF is re-evaluated at each reference, tracking the current field count
of the record as it is modified -- it is not constant per record. This
surprised a user whose C-style for-loop testing i <= NF, with a body
that added fields, became an infinite loop; the docs sentence "their
values ... change from one record to the next" implied per-record
constancy.
Changes:
- reference-dsl-variables.md.in: add an explicit paragraph and live
example showing NF changing mid-expression as fields are added and
removed, plus a caution about for-loops bounded by NF whose bodies
add fields, with the two idiomatic alternatives (snapshot NF first,
or a key-value for-loop over $* which iterates over a copy).
- reference-dsl-control-structures.md.in: note in the while/do-while
section that those examples rely on NF's dynamism, and add a
triple-for note warning about non-terminating NF-bounded loops.
- glossary.md.in: note dynamism in the NF entry.
- pkg/dsl/cst/keyword_usage.go: extend 'mlr help keyword NF' text
(flows into the manpage and the usage-keywords docs section).
- Also fix a pre-existing typo nearby: "if a field has 5 records" ->
"if a record has 5 fields".
- Regenerated docs and man pages.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds a new verb 'mlr bootstrap-ci' which computes bootstrap confidence
intervals for stats1-style statistics: values of the requested fields are
resampled with replacement -n times (default 1000), the statistic is computed
on each resample, and the confidence interval is taken from percentiles of
the resampled statistics at the requested confidence level (-c, default
0.95). For each value field and statistic it emits {field}_{stat} (the
full-data point estimate) along with {field}_{stat}_lo and {field}_{stat}_hi,
optionally grouped by -g. Results are reproducible with mlr --seed.
Includes unit tests, regression-test cases (test/cases/verb-bootstrap-ci),
and documentation updates with regenerated man/doc pages.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add named profile sections to .mlrrc, selected via --profile / -P (#358)
.mlrrc files may now contain INI-style named sections ("profiles"):
# Global settings, applied always:
icsv
[j]
ojson
jvstack
[tsvout]
otsv
Lines before any section header are global settings, applied always, so
existing .mlrrc files behave exactly as before. The new main flag
--profile {name} (alias -P {name}) applies the settings from the [name]
section after the global settings; without it, sections are ignored
entirely (their lines aren't even parsed).
It's a fatal error if a requested profile has no matching section in any
.mlrrc file processed, if no .mlrrc file was found at all, or if
--profile is combined with --norc or MLRRC=__none__.
Also fixes the regression-tester to restore per-case environment
variables to their prior values after each case, rather than setting
them to the empty string -- needed so cases pointing MLRRC at a test
file don't clobber the suite-wide MLRRC=__none__ guard.
Includes unit tests, regression-test cases, and doc/man-page updates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Reject --profile / -P within .mlrrc files (#358)
Per maintainer feedback on the PR: profiles are selected on the mlr
command line, not from within a .mlrrc file, so --profile / -P inside a
.mlrrc file is now a parse error -- the same way --prepipe is rejected
there -- rather than being silently ignored.
Adds a unit test, a should-fail regression case, and a docs bullet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Allow step -a shift_lag/shift_lead/delta/ratio to specify records back/forward (#1002)
Extends the slwin_m_n naming convention to more steppers: shift_lag_12
refers 12 records back, shift_lead_4 refers 4 records forward, and
likewise delta_N, ratio_N, and shift_N. The unsuffixed forms keep their
existing 1-record-back/forward behavior and output field names.
Backward-looking steppers cache copies of previous values in a
fixed-size ring buffer (tValueRing) in their own state, following the
existing pattern of not reading from the window keeper's already-emitted
records, to avoid races with downstream transformers. shift_lead_N uses
the existing forward-window machinery shared with slwin.
Also adds nil guards to all steppers' process functions for the case
where the record at the window center lacks the stepped field, which
previously panicked (pre-existing on main, e.g.
"mlr step -a delta,shift_lead -f x" or "-a slwin_1_1" over
heterogeneous data), and which the new parameterized combinations make
easier to reach.
Includes unit tests, regression cases 0024-0027, and regenerated help
text / docs / man page.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Clarify step -a help text: both plain and _{n} stepper forms are valid (#1002)
Per review on #2190: descriptions like "E.g. delta_7 for 7 records
back" read as if the suffixed form were the only syntax. Reword the
shift, shift_lag, shift_lead, delta, and ratio descriptions to state
both forms explicitly -- e.g. "Use delta or equivalently delta_1 for
the previous record, or delta_{n} for n records back" -- using the {n}
placeholder style, and update the matching usage paragraph and
reference-verbs prose. Regenerate help-derived docs, man page, and
cli-help regression expectation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Give a specific error for invalid stepper count suffixes (#1002)
Per review on #2190: "mlr step -a delta_0" (or delta_-1, delta_x, etc.)
previously reported the generic 'stepper "delta_0" not found'. Now,
when the name prefixes a known count-suffix stepper (shift, shift_lag,
shift_lead, delta, ratio) but the suffix is not a positive integer, the
error is:
mlr step: stepper "delta_0": count must be a positive integer
Truly unknown stepper names keep the existing "not found" message.
Adds should-fail regression cases for delta_0 and delta_-1, and unit
tests for the detection helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add sliding-window mode to stats1 via -w {n} (#1017)
This adds a generic sliding-window option to the stats1 verb, per the
feature request in #1017. With 'mlr stats1 ... -w {n}', statistics are
computed over a trailing window of up to n records -- per group when -g
is used -- and one output record is emitted per input record, with the
windowed statistics appended to it.
The implementation retains, per grouping key, copies of only the
relevant value fields from the last up-to-n records. On each input
record the group's accumulators are reset and re-fed from the window.
This is O(window size) per record, but is fully generic: it works
uniformly for every stats1 accumulator, including order-sensitive ones
(mode, antimode) and non-invertible ones (percentiles, distinct_count),
and composes with -g, --fr/--fx, and --gr/--gx.
Includes unit tests, regression cases (test/cases/verb-stats1/0020-0025),
and regenerated docs/man/help-text artifacts.
Addresses #1017.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* neaten
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds a section to the questions-about-joins page showing how to update
matching records of a durable CSV file from a download containing a
subset of records and a subset of columns, using join --ul with the
database as the left file (right-file values win on collision), plus
variants for restoring row order and for keeping database values when
the download has empty cells.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add a "Transposing very wide data" section to the shapes-of-data page,
showing two existing-verb approaches for viewing wide CSVs rotated 90
degrees: XTAB output format for a per-record vertical view, and a DSL
out-of-stream-variable recipe (thanks @Fravadona) for a full transpose
where each input column becomes an output row.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The workflow pins all three codeql-action steps to the same commit SHA, but
Dependabot was opening separate PRs per action path, so merging just one
broke CI with a CodeQL config-version mismatch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Miller's strftime delegates to the full-featured lestrrat-go/strftime
library, but strptime uses an in-tree fork of a small subset-only
package, so strftime output couldn't always be parsed back by
strptime with the same format string. This adds the format codes
needed to round-trip common formats like "%a %b %e %T %Y":
* %a / %A (weekday name) and %h (alias of %b): straightforward
formatMap entries, same pattern as the existing %b/%B.
* %e (space-padded day of month): needed dedicated width-detection
logic, since %e's own optional leading pad space is otherwise
indistinguishable from a literal separator space when the code
searches for the next literal text to bound a field -- this was
the root cause of the round-trip failure in the linked issue.
* %c, %x, %X: shorthand aliases (expanding to %a %b %e %H:%M:%S %Y,
%m/%d/%y, and %H:%M:%S respectively), same mechanism already used
for %T/%D/%F/%R/%r.
Also documents the %D/%F/%r/%R/%T shorthands in the strptime table
in reference-dsl-time.md.in, which were already supported but
missing from the docs -- this was the exact confusion reported in
the discussion linked from #1518.
Round-trip tests added in pkg/bifs/datetime_test.go assert
strptime(strftime(t, fmt), fmt) == t across the newly-supported
formats, plus table-driven cases in
pkg/pbnjay-strptime/strptime_test.go covering %e's edge cases
(padded/unpadded single digit, double digit, adjacent to another
code with no separator, at end of string).
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* stats1: add rank accumulator (#383)
Adds `mlr stats1 -a rank` for standard competition ranking (1,2,2,4,...)
on pre-sorted data, most useful with -s for a rank on every record.
* stats1: make rank order-independent by default, add --rank-sorted opt-in fast path
The rank accumulator previously only compared each value to the immediately
preceding record, silently giving wrong ranks for non-adjacent duplicates
(e.g. unsorted input, or interleaved -g groups). Default to correctly
computing standard competition rank from all values seen so far
(order-independent, buffers values, same approach as percentile
accumulators). Add --rank-sorted for callers who can promise sorted input
and want the previous O(1)-space streaming behavior instead.
* Revert "stats1: make rank order-independent by default, add --rank-sorted opt-in fast path"
This reverts commit aa45a591fe.
* Revert "stats1: add rank accumulator (#383)"
This reverts commit 96deed048a.
* Add mlr rank verb (#383)
Reverts the earlier stats1 -a rank / --rank-sorted approach: stats1 is a
reduce verb (many records -> one summary record per group) and rank is a
per-record annotator, so it never fit cleanly there -- it needed stats1's
-s iterative-stats escape hatch just to be useful, plus a bolted-on
sorted/unsorted split.
mlr rank is a dedicated verb modeled on mlr fraction: -f fields to rank,
-g optional group-by, output field <f>_rank. By default it's a two-pass
algorithm (buffers input, like fraction does) giving standard competition
rank (1,2,2,4,...) that's correct regardless of input order. --sorted
opts into a single-pass, O(1)-space streaming alternative for callers who
can promise pre-sorted input (e.g. via 'mlr sort' first).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate docs/man pages after merging main (sparkline verb)
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Adds a sparkline(array|map) built-in that renders numeric values as a
Unicode block-character string (▁▂▃▄▅▆▇█), plus a new `mlr sparkline`
verb that summarizes each field's values in record order -- useful for
eyeballing trends without external plotting tools.
Also adds `-s` to `mlr histogram` to sparkline a field's binned counts
(its distribution shape) rather than emitting one record per bin. This
is a different chart from `mlr sparkline` (order-independent binning
vs. record-order values), and the docs for each cross-reference the
other to avoid conflating them.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Discovered while debugging a lint-CI failure on #2166: golangci-lint-action's
persistent cross-commit cache can serve stale staticcheck (SA5011) results
unrelated to a given diff, even when a fresh local run is clean. The job is
continue-on-error so it doesn't block merges, but it's worth documenting so
contributors don't chase a phantom local repro.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Publish an epub of the docs on Read the Docs (#1835)
Read the Docs' built-in formats support (the existing formats: all in
.readthedocs.yaml) only produces epub/PDF for Sphinx projects, and is a
silent no-op for MkDocs ones. Instead, per RTD's documented
build-customization path, generate the epub ourselves in a post_build
job and place it in $READTHEDOCS_OUTPUT/epub/, which RTD then publishes
on the project Downloads page and in the docs flyout menu.
The epub itself is built by the new docs/build-epub.sh: it takes the
committed, generated Markdown pages in docs/src in mkdocs.yml nav
order, strips the HTML-only quicklinks header from each page, and runs
pandoc (installed on RTD via build.apt_packages). Locally, `make -C
docs epub` does the same for anyone with pandoc installed; nothing here
is part of `make dev` or any default build path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix misrender
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The DKVPX record reader set dkvpxReader.Comma = ',' unconditionally,
ignoring the resolved --ifs value; the key-value separator '=' was
likewise hard-coded in the dkvpx parser, ignoring --ips. So e.g.
printf 'a=1;b=2\n' | mlr -i dkvpx --ifs semicolon --ojson cat
produced a single field {"a": "1;b=2"} instead of {"a": 1, "b": 2}.
Fixes:
* pkg/dkvpx.Reader gains an Equals field (default '=') alongside Comma.
* The DKVPX record reader now passes the resolved IFS/IPS through,
validating that each is a single character (mirroring the CSV
reader's "IFS can only be a single character" error) rather than
silently ignoring or truncating them.
* The DKVPX record writer already emitted OFS/OPS, but its quoting
decisions were based on the default separators, so with --ofs
semicolon a value containing ';' went out unquoted (ambiguous) while
a value containing ',' was quoted needlessly. Quoting now keys off
the actual OFS/OPS.
Adds unit tests, regression cases under test/cases/io-dkvpx/, and a
note in the separators reference table.
Discovered while investigating #369; tracked in #2170.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The join verb's left-file ingest silently swallowed read errors: a
nonexistent or malformed left file produced empty (or partial) output
with exit code 0.
Two defects, one per ingest path:
* Unsorted (half-streaming) path, ingestLeftFile in
pkg/transformers/join.go: errors from the record-reader's error
channel were dropped ("TODO: propagate error to caller"), as were
record-reader construction errors. All left-file read failures were
silent.
* Sorted (-s) path, readRecord in
pkg/transformers/utils/join_bucket_keeper.go: the error channel was
consulted, but the record-reader goroutine sends the error and the
end-of-stream marker on separate channels, so the select could see
end-of-stream first and drop the error. A nonexistent left file with
-s exited 0 about half the time (racy).
Both paths now print "mlr: <error>" to stderr and exit 1, matching how
read errors on regular (right) input files are reported. On receipt of
the end-of-stream marker, both consumers now do a final non-blocking
drain of the error channel, which is deterministic since the reader
sends any error before the marker.
Discovered while verifying #377.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The behavior reported in #999 matches Go's encoding/csv LazyQuotes
semantics (and Python's csv module): a field beginning with an unmatched
double quote is a quoted field whose contents extend across field
separators and newlines until the next quote character or end of file.
Add a docs section explaining what --lazy-quotes does and does not
relax, with a worked example, and point to CSV-lite as the alternative
when quote characters are ordinary data.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds a section to the questions-about-joins page showing how to merge
N files side-by-side on a shared key when the files' non-key column
names are all the same: a then-chain of join verbs with per-file --lp
prefixes (plus --lk / cut / label to trim, and --ul --ur / unsparsify
for outer-join semantics). Addresses #1775.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add a section to the record-heterogeneity page showing how to
concatenate multiple CSV files whose headers differ, using unsparsify
(and unsparsify -f then regularize for the streaming case), with new
sample data files under docs/src/data/het/.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
