* Add sliding-window mode to stats1 via -w {n} (#1017)
This adds a generic sliding-window option to the stats1 verb, per the
feature request in #1017. With 'mlr stats1 ... -w {n}', statistics are
computed over a trailing window of up to n records -- per group when -g
is used -- and one output record is emitted per input record, with the
windowed statistics appended to it.
The implementation retains, per grouping key, copies of only the
relevant value fields from the last up-to-n records. On each input
record the group's accumulators are reset and re-fed from the window.
This is O(window size) per record, but is fully generic: it works
uniformly for every stats1 accumulator, including order-sensitive ones
(mode, antimode) and non-invertible ones (percentiles, distinct_count),
and composes with -g, --fr/--fx, and --gr/--gx.
Includes unit tests, regression cases (test/cases/verb-stats1/0020-0025),
and regenerated docs/man/help-text artifacts.
Addresses #1017.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* neaten
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds a section to the questions-about-joins page showing how to update
matching records of a durable CSV file from a download containing a
subset of records and a subset of columns, using join --ul with the
database as the left file (right-file values win on collision), plus
variants for restoring row order and for keeping database values when
the download has empty cells.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add a "Transposing very wide data" section to the shapes-of-data page,
showing two existing-verb approaches for viewing wide CSVs rotated 90
degrees: XTAB output format for a per-record vertical view, and a DSL
out-of-stream-variable recipe (thanks @Fravadona) for a full transpose
where each input column becomes an output row.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The workflow pins all three codeql-action steps to the same commit SHA, but
Dependabot was opening separate PRs per action path, so merging just one
broke CI with a CodeQL config-version mismatch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Miller's strftime delegates to the full-featured lestrrat-go/strftime
library, but strptime uses an in-tree fork of a small subset-only
package, so strftime output couldn't always be parsed back by
strptime with the same format string. This adds the format codes
needed to round-trip common formats like "%a %b %e %T %Y":
* %a / %A (weekday name) and %h (alias of %b): straightforward
formatMap entries, same pattern as the existing %b/%B.
* %e (space-padded day of month): needed dedicated width-detection
logic, since %e's own optional leading pad space is otherwise
indistinguishable from a literal separator space when the code
searches for the next literal text to bound a field -- this was
the root cause of the round-trip failure in the linked issue.
* %c, %x, %X: shorthand aliases (expanding to %a %b %e %H:%M:%S %Y,
%m/%d/%y, and %H:%M:%S respectively), same mechanism already used
for %T/%D/%F/%R/%r.
Also documents the %D/%F/%r/%R/%T shorthands in the strptime table
in reference-dsl-time.md.in, which were already supported but
missing from the docs -- this was the exact confusion reported in
the discussion linked from #1518.
Round-trip tests added in pkg/bifs/datetime_test.go assert
strptime(strftime(t, fmt), fmt) == t across the newly-supported
formats, plus table-driven cases in
pkg/pbnjay-strptime/strptime_test.go covering %e's edge cases
(padded/unpadded single digit, double digit, adjacent to another
code with no separator, at end of string).
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* stats1: add rank accumulator (#383)
Adds `mlr stats1 -a rank` for standard competition ranking (1,2,2,4,...)
on pre-sorted data, most useful with -s for a rank on every record.
* stats1: make rank order-independent by default, add --rank-sorted opt-in fast path
The rank accumulator previously only compared each value to the immediately
preceding record, silently giving wrong ranks for non-adjacent duplicates
(e.g. unsorted input, or interleaved -g groups). Default to correctly
computing standard competition rank from all values seen so far
(order-independent, buffers values, same approach as percentile
accumulators). Add --rank-sorted for callers who can promise sorted input
and want the previous O(1)-space streaming behavior instead.
* Revert "stats1: make rank order-independent by default, add --rank-sorted opt-in fast path"
This reverts commit aa45a591fe.
* Revert "stats1: add rank accumulator (#383)"
This reverts commit 96deed048a.
* Add mlr rank verb (#383)
Reverts the earlier stats1 -a rank / --rank-sorted approach: stats1 is a
reduce verb (many records -> one summary record per group) and rank is a
per-record annotator, so it never fit cleanly there -- it needed stats1's
-s iterative-stats escape hatch just to be useful, plus a bolted-on
sorted/unsorted split.
mlr rank is a dedicated verb modeled on mlr fraction: -f fields to rank,
-g optional group-by, output field <f>_rank. By default it's a two-pass
algorithm (buffers input, like fraction does) giving standard competition
rank (1,2,2,4,...) that's correct regardless of input order. --sorted
opts into a single-pass, O(1)-space streaming alternative for callers who
can promise pre-sorted input (e.g. via 'mlr sort' first).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate docs/man pages after merging main (sparkline verb)
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Adds a sparkline(array|map) built-in that renders numeric values as a
Unicode block-character string (▁▂▃▄▅▆▇█), plus a new `mlr sparkline`
verb that summarizes each field's values in record order -- useful for
eyeballing trends without external plotting tools.
Also adds `-s` to `mlr histogram` to sparkline a field's binned counts
(its distribution shape) rather than emitting one record per bin. This
is a different chart from `mlr sparkline` (order-independent binning
vs. record-order values), and the docs for each cross-reference the
other to avoid conflating them.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Discovered while debugging a lint-CI failure on #2166: golangci-lint-action's
persistent cross-commit cache can serve stale staticcheck (SA5011) results
unrelated to a given diff, even when a fresh local run is clean. The job is
continue-on-error so it doesn't block merges, but it's worth documenting so
contributors don't chase a phantom local repro.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Publish an epub of the docs on Read the Docs (#1835)
Read the Docs' built-in formats support (the existing formats: all in
.readthedocs.yaml) only produces epub/PDF for Sphinx projects, and is a
silent no-op for MkDocs ones. Instead, per RTD's documented
build-customization path, generate the epub ourselves in a post_build
job and place it in $READTHEDOCS_OUTPUT/epub/, which RTD then publishes
on the project Downloads page and in the docs flyout menu.
The epub itself is built by the new docs/build-epub.sh: it takes the
committed, generated Markdown pages in docs/src in mkdocs.yml nav
order, strips the HTML-only quicklinks header from each page, and runs
pandoc (installed on RTD via build.apt_packages). Locally, `make -C
docs epub` does the same for anyone with pandoc installed; nothing here
is part of `make dev` or any default build path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix misrender
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The DKVPX record reader set dkvpxReader.Comma = ',' unconditionally,
ignoring the resolved --ifs value; the key-value separator '=' was
likewise hard-coded in the dkvpx parser, ignoring --ips. So e.g.
printf 'a=1;b=2\n' | mlr -i dkvpx --ifs semicolon --ojson cat
produced a single field {"a": "1;b=2"} instead of {"a": 1, "b": 2}.
Fixes:
* pkg/dkvpx.Reader gains an Equals field (default '=') alongside Comma.
* The DKVPX record reader now passes the resolved IFS/IPS through,
validating that each is a single character (mirroring the CSV
reader's "IFS can only be a single character" error) rather than
silently ignoring or truncating them.
* The DKVPX record writer already emitted OFS/OPS, but its quoting
decisions were based on the default separators, so with --ofs
semicolon a value containing ';' went out unquoted (ambiguous) while
a value containing ',' was quoted needlessly. Quoting now keys off
the actual OFS/OPS.
Adds unit tests, regression cases under test/cases/io-dkvpx/, and a
note in the separators reference table.
Discovered while investigating #369; tracked in #2170.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The join verb's left-file ingest silently swallowed read errors: a
nonexistent or malformed left file produced empty (or partial) output
with exit code 0.
Two defects, one per ingest path:
* Unsorted (half-streaming) path, ingestLeftFile in
pkg/transformers/join.go: errors from the record-reader's error
channel were dropped ("TODO: propagate error to caller"), as were
record-reader construction errors. All left-file read failures were
silent.
* Sorted (-s) path, readRecord in
pkg/transformers/utils/join_bucket_keeper.go: the error channel was
consulted, but the record-reader goroutine sends the error and the
end-of-stream marker on separate channels, so the select could see
end-of-stream first and drop the error. A nonexistent left file with
-s exited 0 about half the time (racy).
Both paths now print "mlr: <error>" to stderr and exit 1, matching how
read errors on regular (right) input files are reported. On receipt of
the end-of-stream marker, both consumers now do a final non-blocking
drain of the error channel, which is deterministic since the reader
sends any error before the marker.
Discovered while verifying #377.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The behavior reported in #999 matches Go's encoding/csv LazyQuotes
semantics (and Python's csv module): a field beginning with an unmatched
double quote is a quoted field whose contents extend across field
separators and newlines until the next quote character or end of file.
Add a docs section explaining what --lazy-quotes does and does not
relax, with a worked example, and point to CSV-lite as the alternative
when quote characters are ordinary data.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds a section to the questions-about-joins page showing how to merge
N files side-by-side on a shared key when the files' non-key column
names are all the same: a then-chain of join verbs with per-file --lp
prefixes (plus --lk / cut / label to trim, and --ul --ur / unsparsify
for outer-join semantics). Addresses #1775.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add a section to the record-heterogeneity page showing how to
concatenate multiple CSV files whose headers differ, using unsparsify
(and unsparsify -f then regularize for the streaming case), with new
sample data files under docs/src/data/het/.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Add a worked-examples section to the questions-about-joins page showing
how to get SQL-style join semantics (keeping non-matching records) with
mlr join --ul/--ur followed by unsparsify, using the example data from
issue #652.
Resolves#652.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds a recipe section to the questions-about-joins page showing two
ways to preprocess the join verb's -f left file without creating an
intermediate file: the main-level --prepipe flag (which the left file
inherits along with the other main input options), and shell process
substitution. Also notes that the verb-level --prepipe/--prepipex
flags listed in 'mlr join --help' do not currently take effect for
the left file in Miller 6.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Documents the "TZ environment variable appears malformed" / "time:
invalid location name" startup errors: TZ set to a file path (which
Go's time library rejects, unlike C), and missing timezone databases
on Windows, Cygwin, and minimal containers. Lists remedies: IANA zone
names, installing tzdata, the ZONEINFO environment variable, and
fixed-offset formats.
Addresses #1238; also relevant to #1406 and #1758.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
With --right-align-numeric, PPRINT data cells right-align but headers
stayed left-aligned, so a header did not line up with its own column's
data -- the original ask in #380. Now a header is right-aligned when
every value in its column is numeric, for both non-barred and barred
PPRINT output. Mixed columns keep left-aligned headers.
For --omd-aligned, the raw header text of right-aligned columns is now
right-justified too, matching how Markdown viewers render the ---:
marker; this follows the same all-values-numeric per-column rule
already used for the separator markers.
Man-page regeneration also picks up previously-merged reorder help-text
edits that had not been regenerated.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The uniq verb outputs only the group-by columns, and issue #1075 asks
for a way to deduplicate on some fields while keeping the rest. Miller
already supports this via "head -n 1 -g", but that wasn't discoverable
from uniq's help text or its reference-verbs section. Add a note to
"mlr uniq --help" and a short recipe (with live examples) to the uniq
section of the verbs reference, and regenerate the derived man page,
docs, and CLI-help golden files.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
When the skip-trivial-records verb is in the then-chain, the CSV/TSV
record-readers now silently skip trivial (all-fields-empty) input lines
-- notably blank lines at the end of a CSV file -- instead of raising a
fatal header/data length mismatch. The user has explicitly asked for
trivial records to be skipped, so mlr now exits 0 in this case.
The verb's CLI parser sets a new ReaderOptions.SkipTrivialRecords flag,
which the CSV and TSV readers consult only on the would-be-fatal
mismatch path, so:
* Behavior without the verb is unchanged: blank lines still error.
* Genuinely ragged non-trivial records still error even with the verb.
* --allow-ragged-csv-input behavior is unchanged.
Fixes#1535.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
In the explicit-header ragged code paths of the csvlite, TSV, and
barred-pprint readers, the header-longer-than-data fill used
record.PutCopy with the un-deduplicated header name. With duplicate
header names (e.g. "name,4,4" deduped to "name,4,4_2"), the VOID fill
for the missing trailing field overwrote the already-stored value under
the colliding key, so a short row like "bar,sea" lost "sea".
Use record.PutReferenceMaybeDedupe, as the implicit-header paths
already do (where PR #794 had applied it, although implicit headers are
generated unique and never need dedupe).
Adds regression cases under test/cases/io-dedupe-field-names/ for the
csvlite, TSV, and barred-pprint readers.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Plain {} placeholders consume arguments sequentially as before; {N}
refers to the Nth argument (1-based) and may be used to repeat and/or
reorder arguments, e.g. format("{1}/{2}/{1}_{3}.ext", $p1, $p2, $p3).
Mixing is allowed: the {} sequence counter is independent of positional
placeholders (as in Rust format!). Out-of-range indices interpolate the
empty string, consistent with existing too-few-arguments behavior; {0}
is an error value since indices are 1-based.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
With multiple regexes, 'mlr reorder -r' previously emitted all matched
fields in record order, ignoring the order the regexes were given. Now
matched fields are grouped by regex-list order (first regex's matches
first, then the second's, etc.); within each group, fields keep their
record order. A field matching multiple regexes is claimed by the first
one. This makes 'reorder -r' consistent with 'cut -orf' and satisfies
the original request in #1325: -r '^YYY,^XXX' puts YYY-prefixed fields
first, then XXX-prefixed fields, then the rest. Applies to -e, -b, and
-a modes as well.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
For PPRINT, right-justifies data cells whose values are numeric,
leaving other cells and header lines left-justified. For Markdown,
emits right-alignment markers (---:) for numeric columns; in
--omd-aligned mode also right-justifies the raw cell text.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Honor --ors CRLF for CSV output (#1810)
The full-CSV record writer validated ORS as newline or
carriage-return/newline, but never propagated the choice to the forked
Go CSV writer's UseCRLF field, so `--ors '\r\n'` (or `--ors crlf`)
silently produced LF line endings. Set UseCRLF from the writer options
so CRLF output is honored. Default behavior (LF) is unchanged, and
other ORS values are still rejected. The CSV-lite and TSV writers
already honored CRLF ORS.
Also: fix a copy-paste "for CSV" in the TSV writer's ORS-validation
message, add unit tests asserting byte-exact line endings (the regtest
harness normalizes CR/LF, so this can't be asserted in test/cases), add
CLI-level regression cases, and update the separators documentation.
This substantially addresses #1722 as well: RFC-4180-style CRLF output
can now be requested on any platform.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix errcheck lint: check Flush() error in CSV writer test
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The summary verb's field_type column shows values like "string-int" or
"empty-string" for columns containing values of mixed inferred types:
all types encountered across records are printed, hyphen-joined, in the
order first encountered. Add a note to the verb's help text explaining
this, and regenerate the man page, docs, and golden test output.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Miller type-inference accepts float formats which the JSON grammar
disallows -- leading zeros like 004.56, leading plus signs, bare leading
or trailing decimal points like .56 or 4., etc. -- and the JSON writer
passed the original text through verbatim, producing invalid JSON.
Leading-zero integers such as 0123456789 were already handled (inferred
as strings), and hex/binary/octal ints were already re-rendered as
decimal for JSON per #1761; this does the same for floats: when a
float string representation is not a valid JSON number, re-render it
(e.g. 004.56 -> 4.56).
Also documents in reference-main-data-types that CSV double-quoting
does not affect type inference, with -S / --infer-none and string()
as the ways to keep such values as strings.
Addresses #1114 and #1293.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The tee verb passed the same record pointer both to its file-output
handler (which writes asynchronously on another goroutine, and whose
buffering formats like pprint/json can hold records until close) and
downstream in the verb chain, where subsequent verbs mutate records in
place. Downstream mutations could therefore leak into the tee'd output:
mlr tee -p cat then cat -n then nothing <<EOF
a=1,b=2
a=3,b=4
a=5,b=6
EOF
emitted "n=3,a=5,b=6" for the last tee'd record, and with pprint/json
tee formats every record picked up downstream fields.
Fix: give the file-writer its own deep copy of the record. Same fix
applied to the split verb when -v (emit downstream) is used, which
shared records with downstream the same way.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The join verb's help text listed only '-i {one of csv,dkvp,nidx,pprint,xtab}'
for overriding the left-file input format, but the verb also accepts the
--icsv/--ijson-style main-flag shorthands (and formats beyond the five
listed, e.g. json and tsv), since unrecognized verb flags fall through to
the main flag table. Update the usage text to say so, and regenerate the
man page and docs content that embed this help output.
Fixes#444.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The uniq verb writes its group-by fields in the order they are named
with -g or -f, not in the order they appear in the input records
(unlike cut, which preserves input order unless -o is given). Per
discussion on #962, document this in the verb help text rather than
change long-standing output ordering. Regenerate the man page and the
docs pages which embed the verb help.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
