* Publish an epub of the docs on Read the Docs (#1835)
Read the Docs' built-in formats support (the existing formats: all in
.readthedocs.yaml) only produces epub/PDF for Sphinx projects, and is a
silent no-op for MkDocs ones. Instead, per RTD's documented
build-customization path, generate the epub ourselves in a post_build
job and place it in $READTHEDOCS_OUTPUT/epub/, which RTD then publishes
on the project Downloads page and in the docs flyout menu.
The epub itself is built by the new docs/build-epub.sh: it takes the
committed, generated Markdown pages in docs/src in mkdocs.yml nav
order, strips the HTML-only quicklinks header from each page, and runs
pandoc (installed on RTD via build.apt_packages). Locally, `make -C
docs epub` does the same for anyone with pandoc installed; nothing here
is part of `make dev` or any default build path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix misrender
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
With --right-align-numeric, PPRINT data cells right-align but headers
stayed left-aligned, so a header did not line up with its own column's
data -- the original ask in #380. Now a header is right-aligned when
every value in its column is numeric, for both non-barred and barred
PPRINT output. Mixed columns keep left-aligned headers.
For --omd-aligned, the raw header text of right-aligned columns is now
right-justified too, matching how Markdown viewers render the ---:
marker; this follows the same all-values-numeric per-column rule
already used for the separator markers.
Man-page regeneration also picks up previously-merged reorder help-text
edits that had not been regenerated.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Plain {} placeholders consume arguments sequentially as before; {N}
refers to the Nth argument (1-based) and may be used to repeat and/or
reorder arguments, e.g. format("{1}/{2}/{1}_{3}.ext", $p1, $p2, $p3).
Mixing is allowed: the {} sequence counter is independent of positional
placeholders (as in Rust format!). Out-of-range indices interpolate the
empty string, consistent with existing too-few-arguments behavior; {0}
is an error value since indices are 1-based.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
For PPRINT, right-justifies data cells whose values are numeric,
leaving other cells and header lines left-justified. For Markdown,
emits right-alignment markers (---:) for numeric columns; in
--omd-aligned mode also right-justifies the raw cell text.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Honor --ors CRLF for CSV output (#1810)
The full-CSV record writer validated ORS as newline or
carriage-return/newline, but never propagated the choice to the forked
Go CSV writer's UseCRLF field, so `--ors '\r\n'` (or `--ors crlf`)
silently produced LF line endings. Set UseCRLF from the writer options
so CRLF output is honored. Default behavior (LF) is unchanged, and
other ORS values are still rejected. The CSV-lite and TSV writers
already honored CRLF ORS.
Also: fix a copy-paste "for CSV" in the TSV writer's ORS-validation
message, add unit tests asserting byte-exact line endings (the regtest
harness normalizes CR/LF, so this can't be asserted in test/cases), add
CLI-level regression cases, and update the separators documentation.
This substantially addresses #1722 as well: RFC-4180-style CRLF output
can now be requested on any platform.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix errcheck lint: check Flush() error in CSV writer test
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The summary verb's field_type column shows values like "string-int" or
"empty-string" for columns containing values of mixed inferred types:
all types encountered across records are printed, hyphen-joined, in the
order first encountered. Add a note to the verb's help text explaining
this, and regenerate the man page, docs, and golden test output.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The join verb's help text listed only '-i {one of csv,dkvp,nidx,pprint,xtab}'
for overriding the left-file input format, but the verb also accepts the
--icsv/--ijson-style main-flag shorthands (and formats beyond the five
listed, e.g. json and tsv), since unrecognized verb flags fall through to
the main flag table. Update the usage text to say so, and regenerate the
man page and docs content that embed this help output.
Fixes#444.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The uniq verb writes its group-by fields in the order they are named
with -g or -f, not in the order they appear in the input records
(unlike cut, which preserves input order unless -o is given). Per
discussion on #962, document this in the verb help text rather than
change long-standing output ordering. Regenerate the man page and the
docs pages which embed the verb help.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* docs: add "Miller and AI agents" quick-start page (#2098)
Umbrella page for the AI-friendly feature stack: one-line MCP setup as
the fast path, plus the plain-CLI path (which, help --as-json,
describe, --explain, --errors-json, --no-shell) with live CI-tested
examples, and the discover -> constrain -> validate -> run loop.
Cross-linked with the MCP server page; listed under Getting started.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: reframe AI page as "Miller and AI" (#2098)
Rename ai-agents.md to ai.md and restructure around the pre-MCP
feature stack, organized as the loop each feature serves: Discover
(catalog/index/which, cache keys, single-sourced usage text),
Constrain (enum value-sets + describe: tool shape vs data shape),
Validate (--explain), Run and recover (--errors-json, --no-shell,
env-var trio). MCP is now one closing section pointing at the
mcp-server.md detail page. All examples are live and CI-tested,
including Miller querying its own catalog.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: retitle MCP page to "The MCP server" under the Miller-and-AI umbrella
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: add bare-minimum getting-started section to Miller-and-AI page
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: AI features land in Miller 6.20
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: hyperlink SKILL.md references to the repo
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: rename section to 'The essentials'
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Plan: flesh out PR7 (MCP server + Agent Skill) design
stdio transport (no HTTP port), mlr mcp terminal in the main binary,
SDK-vs-handroll decision, tool list, in-process vs subprocess split,
run-tool safety (--no-shell prerequisite), single-sourced skill, tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add mlr mcp: MCP server + agent playbook; --no-shell gate (#2098 PR7)
New terminal `mlr mcp` runs a Model Context Protocol server over stdio
(spawned by MCP clients; no network port), exposing five tools --
list_capabilities, which, validate_dsl, describe_data, run -- plus an
agent playbook as MCP prompt/resource. Catalog tools are served
in-process from the help registries; the rest subprocess this same
binary with MLR_ERRORS_JSON=1, a timeout, and an output cap.
Prerequisite: a new --no-shell flag / MLR_NO_SHELL env var (one-way
gate) disables the DSL system/exec functions, piped redirects, and
--prepipe/--prepipex; the MCP server sets it on the commands it runs
unless started with --allow-shell.
Adds the github.com/modelcontextprotocol/go-sdk dependency.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Force LF checkout for the embedded SKILL.md (Windows CI fix)
go:embed embeds checkout bytes, so a CRLF checkout on Windows made the
embedded playbook differ per platform and failed
TestPlaybookHasFrontmatter. Pin the file to eol=lf in .gitattributes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Move no-shell test DSL into per-case mlr files (Windows CI fix)
Inline single-quoted DSL in cmd files is mangled by the Windows shell
(single quotes are not quote characters there); the harness's
put -f ${CASEDIR}/mlr pattern avoids shell quoting entirely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
One output record per input field: types seen with counts, occurrence
count, null count, cardinality, min/max, and -- for fields within the
-n/--max-values cap -- the complete distinct-value list in first-seen
order. `mlr --ojson describe` is the machine-readable form; nested
types/values flatten in tabular formats.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Lets an agent type-check a DSL expression before spending a full input
pass. `mlr put --explain '...'` (and filter) runs the existing
parse -> ValidateAST -> CST build -> Resolve path, then:
- valid: prints "mlr {put,filter}: DSL expression is valid." and exits 0
- invalid: returns the build error up the normal path, so --errors-json
emits a structured document; exits 1
- -W with fatal warnings: reports and exits 1
The gate lives in the pass-two constructor, before any input file is
opened, so no input stream is read (verified with a nonexistent input
file still validating OK).
Also categorize bare "parse error: ..." messages from the DSL parser as
kind "dsl-parse-error" rather than "generic" (climain/errors_json.go),
so --explain --errors-json gives an agent a useful error kind. The CSV
reader's "parse error on line ..." is stream-time and never reaches this
command-line-parse categorizer.
Tests: dsl-explain/0001-0004 regression cases (valid put/filter, invalid
plain, invalid --errors-json) and categorize unit tests. Regenerated
verb docs, manpage, and the help usage-verbs golden case.
The older -X ("exit after parsing") still exits 0 even on a parse error;
left as-is since --explain is the correct validation path.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add MT_BYTES mlrval type: foundation and disposition tables
First step toward a first-class bytes type in the DSL (#1231).
Adds MT_BYTES (payload []byte, rendered as lowercase hex in all output
formats, JSON-encoded as a hex string), extends every disposition
matrix/vector with the new row/column -- real cells for comparison,
sorting, and dot-concat of bytes with bytes; type-error stubs
elsewhere -- and adds sweep tests asserting no table has nil cells,
since Go zero-fills short array literals when MT_DIM grows.
Bytes values are not yet constructible from the DSL; b"..." literals
and constructor/codec functions follow in subsequent commits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add b"..." bytes-literal syntax to the DSL
Adds a bytes_literal token to the grammar (regenerating the PGPG lexer
and parser) and a BytesLiteralNode in the CST which evaluates to an
MT_BYTES mlrval. Escape handling reuses UnbackslashStringLiteral,
which is already byte-oriented: b"\xff" is the single byte 0xff.
Unlike string literals, bytes literals never participate in
regex-capture replacement. A bare identifier b is unaffected.
Part of #1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add bytes DSL functions: conversions, codecs, and bytes-aware built-ins
- bytes(x) converts strings to bytes; string(b) reinterprets raw bytes
as UTF-8 text (the reverse)
- base64_decode now always returns bytes (superseding the interim
string-or-hex behavior); base64_encode accepts string or bytes
- New hex_encode/hex_decode functions
- is_bytes and asserting_bytes predicates
- md5/sha1/sha256/sha512 accept bytes, hashing the raw payload
- strlen of bytes is the byte count; substr/substr0/substr1 on bytes
slice by byte position and return bytes
The Cyrillic-LDAP scenario from #1231 now works without exec
workarounds: string(base64_decode($x)) recovers the text, and binary
payloads survive undamaged as bytes.
Closes#1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add bytes-type docs and regression cases
Documents the bytes type on the data-types page, regenerates the
function-reference/man-page material, and adds regression coverage:
literal escape forms, operators (concat/compare/slice/sort and
type errors), conversions and codec round-trips, and CSV-to-JSON
output rendering of bytes fields.
Part of #1231.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Reposition MT_BYTES to sort adjacent to MT_STRING in the type enum
MT_BYTES was appended after MT_ABSENT for index stability; move it
right after MT_STRING instead, since that's where it conceptually
belongs and where it already sorts in the cmp disposition matrices.
Mechanically re-derive all ~40 disposition tables in pkg/bifs and
pkg/mlrval accordingly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix windows CI
* fix merge
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Default to "cat" verb when none is supplied (#2029)
Invocations like 'mlr --j2y' or 'mlr --c2p' previously failed with
"no verb supplied", forcing users to type the trailing 'cat'
explicitly for pure format conversions. Default the verb to 'cat'
in that case. Bare 'mlr' with no flags, no verb, and no files
still prints the main usage banner.
This handles flag-only invocations (e.g. 'mlr --c2j < input.csv'
or 'mlr --c2j --from input.csv'). File names without a preceding
verb are still parsed as verb candidates and continue to error if
not found; that broader change is out of scope here.
* Use ${MLR} substitution for bare-mlr regression case (#2029)
The regtester only substitutes the mlr executable when the cmd
starts with "mlr " (with a trailing space). The bare-mlr usage-banner
test had a cmd of just "mlr", so on CI -- which invokes regtest with
a relative path like 'test/../mlr' -- the test shelled out to a
literal 'mlr' that isn't on PATH and failed with exit 127.
Switch the cmd to ${MLR} (the regtester's explicit substitution
token) so the case runs the right binary in any invocation context.
Adds a new Markdown-only flag --omd-aligned (alias --omarkdown-aligned)
that left-justifies cells and pads each column to a uniform width.
The rendered table is unaffected; the goal is readability of the raw
markdown source. The flag implies --omd, so users do not need to pass
both.
Implementation batches records by schema (like pprint), computes max
column widths, then emits header / separator / data rows with bars
vertically aligned. Default markdown writer behavior is unchanged.
* reader
* record writer
* pkg/output/record_writer_dcf.go
* test/input/test.dcf
* test/cases/io-dcf/
* pkg/cli/option_parse.go
* make fmt
* make dev
* docs and `make dev`
* docs/src/proposal-yaml-io.md
* initial step
* Add `--y2c` etc
* test cases
* docs
* more testing
* more
* more test cases
* git rm docs/src/proposal-yaml-io.md
* make dev
* ylistwrap -> yarray
