Add --errors-json structured error output (#2098) (#2113)

* Tier-2 structured verb options: OptionSpec, initial migration (#2098)

PR 3 of the AI-friendly roadmap (plans/plan-2098-llm.md).

Infrastructure:
- Add OptionSpec{Flag,Arg,Type,Desc,Repeatable,Values} to
  pkg/transformers/aaa_record_transformer.go alongside TransformerSetup.
  Type is one of: bool, string, int, float, csv-list, regex, filename,
  format, enum. For type=="enum", Values lists the valid choices.
- Add Options []OptionSpec to TransformerSetup (nil = not yet migrated).
- Emit Options in VerbInfoForJSON (omitempty so unmigrated verbs stay
  backward-compatible; agents check key presence for Tier-2 availability).
  UsageText is always present as the Tier-1 prose fallback.
- Add VerbOptionsNilCheck() in aaa_verb_options_check.go: progress report
  of migrated vs. unmigrated verbs, analogous to FLAG_TABLE.NilCheck().
- Wire verb-options-nil-check into mlr help (internal/docgen section).

Initial migration (5/70 verbs):
- nothing: empty Options (no verb-specific options, explicitly migrated)
- cat: -n (bool), -N (string), -g (csv-list), --filename, --filenum (bool)
- head: -g (csv-list), -n (int)
- tail: -g (csv-list), -n (int)
- tee: -a, -p (bool)

Tests:
- 5 new unit tests in aaa_transformer_json_test.go covering migrated/
  unmigrated paths, field population, JSON round-trip, and key-presence.
- Regression test case 0003: mlr help verb-options-nil-check golden output.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* Migrate all 70 verbs to structured OptionSpec; bump catalog schema to v2

Completes the Tier-2 migration started in the previous commit. Every verb
in TRANSFORMER_LOOKUP_TABLE now has a non-nil Options field.

- Workflow-migrated all 65 remaining verbs. Each Setup var now carries
  Options: []OptionSpec{...} with Flag/Arg/Type/Desc fields. Verbs with
  no verb-specific options (altkv, check, group-like, nothing, etc.) use
  an empty slice to signal "migrated but no options."
- Drop `omitempty` from VerbInfoForJSON.Options: empty slices were silently
  dropped, making migrated-no-option verbs indistinguishable from unmigrated
  ones in JSON. Without omitempty: null=unmigrated, []=migrated-no-options,
  [...]= migrated-with-options. Bump catalogSchemaVersion 1→2 for this shape
  change.
- Replace the two "unmigrated-verb" unit tests (which used stats1 as an
  example) with TestAllVerbsFullyMigrated (asserts every verb has non-nil
  Options) and TestAllVerbsHaveOptionsKeyInJSON (asserts every migrated
  verb emits the "options" key in JSON).
- Regenerate test/cases/cli-help/0003/expout: now reads
  "Verb options migration: 70/70 migrated."

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

* Add --errors-json structured error output (#2098)

PR 4 of the AI-friendly roadmap (plans/plan-2098-llm.md).

Agents previously had to regex-match English prose to branch on error
kind; this PR lets them do it structurally.

Interface:
- mlr --errors-json <bad command> emits a JSON object to stderr and
  exits 1; same behavior as prose path but machine-readable.
- MLR_ERRORS_JSON=1 (truthy) is the env-var equivalent.
- Without the flag, prose output is byte-for-byte unchanged.

JSON shape: {error, kind, token, verb, hint, did_you_mean[]}
Error kinds: unknown-verb, unknown-flag, verb-option-error, generic.

Implementation:
- New pkg/climain/errors_json.go: CLIError typed error, StructuredError
  DTO, WantErrorsJSON pre-scan, Levenshtein edit distance, topMatches,
  EmitStructuredError.
- Convert two direct os.Exit sites in parseCommandLinePassOne (unknown
  verb, unknown flag) to return CLIError values; verb-option-error
  likewise returns CLIError from pass-one ParseCLIFunc handling.
- parseCommandLinePassOne gains an error return; ParseCommandLine
  propagates it.
- entrypoint.go pre-scans os.Args for --errors-json before calling
  ParseCommandLine; routes errors to EmitStructuredError or printError.
- did_you_mean: Levenshtein nearest-match over verb catalog, flag
  catalog, or the verb own OptionSpec flags (PR3 catalog) for
  verb-option-error. Closes the self-correction loop the catalog
  enables.
- --errors-json registered in Miscellaneous flags so it appears in
  mlr --help and is not treated as an unrecognized flag.

Tests: 16 unit tests covering Levenshtein, topMatches, threshold,
WantErrorsJSON, CLIError interface, and all categorize paths.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
John Kerl 2026-07-03 14:49:48 -04:00 committed by GitHub
parent f637633420
commit 78ed1563db
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 546 additions and 18 deletions

View file

@ -3385,6 +3385,17 @@ var MiscFlagSection = FlagSection{
infoPrinter: MiscPrintInfo,
flags: []Flag{
{
name: "--errors-json",
help: "Emit parse errors as a JSON object to stderr instead of a plain text message. Intended for AI agents and scripts that branch on error kind rather than regex-matching prose. Equivalent to setting the `MLR_ERRORS_JSON` environment variable to a truthy value.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
// The actual effect is handled by pre-scan in pkg/entrypoint.
// Registration here ensures the flag is recognized (not an
// error) during pass-one flag parsing, and appears in --help.
*pargi += 1
},
},
{
name: "-x",
help: "If any record has an error value in it, report it and stop the process. The default is to print the field value as `(error)` and continue.",

315
pkg/climain/errors_json.go Normal file
View file

@ -0,0 +1,315 @@
// Structured error output for `mlr --errors-json`.
//
// When --errors-json is set (or MLR_ERRORS_JSON is truthy), parse-time errors
// are emitted as a JSON object to stderr instead of a plain text message.
// This lets AI agents and scripts branch on error kind rather than regex-
// matching English prose.
//
// JSON shape:
// {
// "error": "mlr: verb \"flitre\" not found",
// "kind": "unknown-verb",
// "token": "flitre",
// "verb": "", // set for verb-option errors
// "hint": "Run 'mlr filter --help' for usage.",
// "did_you_mean": ["filter", "flatten"]
// }
package climain
import (
"encoding/json"
"errors"
"fmt"
"os"
"sort"
"strings"
"github.com/johnkerl/miller/v6/pkg/cli"
"github.com/johnkerl/miller/v6/pkg/dsl/cst"
"github.com/johnkerl/miller/v6/pkg/transformers"
)
// ----------------------------------------------------------------
// CLIError is the typed error returned by the pass-one parser for
// errors that carry structured metadata (unknown verb, unknown flag).
// Other parse errors still flow as plain errors; EmitStructuredError
// handles both.
type CLIError struct {
Kind string // "unknown-verb" | "unknown-flag"
Token string // the unrecognized token
Verb string // enclosing verb context, if any
Msg string // human-readable message (same as what prose path would print)
}
func (e *CLIError) Error() string { return e.Msg }
// ----------------------------------------------------------------
// StructuredError is the JSON DTO emitted to stderr.
type StructuredError struct {
Error string `json:"error"`
Kind string `json:"kind"`
Token string `json:"token,omitempty"`
Verb string `json:"verb,omitempty"`
Hint string `json:"hint,omitempty"`
DidYouMean []string `json:"did_you_mean,omitempty"`
}
// ----------------------------------------------------------------
// Opt-in detection
// WantErrorsJSON returns true when the caller has opted in via the
// --errors-json flag anywhere in args, or a truthy MLR_ERRORS_JSON env var.
// It is called before ParseCommandLine so it can affect how parse errors
// are reported.
func WantErrorsJSON(args []string) bool {
if isTruthyEnv(os.Getenv("MLR_ERRORS_JSON")) {
return true
}
for _, arg := range args {
if arg == "--errors-json" {
return true
}
}
return false
}
func isTruthyEnv(v string) bool {
switch v {
case "1", "true", "True", "TRUE", "yes", "Yes", "YES":
return true
}
return false
}
// ----------------------------------------------------------------
// Error categorization and emission
// EmitStructuredError writes a JSON error document to stderr and exits 1.
// It is called in place of printError when --errors-json is active.
func EmitStructuredError(err error) {
se := categorize(err)
se.DidYouMean = nearMatches(se)
b, jerr := json.MarshalIndent(se, "", " ")
if jerr != nil {
// Fallback: plain text if we somehow can't marshal
fmt.Fprintf(os.Stderr, "mlr: %v\n", err)
return
}
fmt.Fprintln(os.Stderr, string(b))
}
// categorize turns any error into a StructuredError, using the typed CLIError
// fields when available and string-matching as a fallback.
func categorize(err error) StructuredError {
var cliErr *CLIError
if errors.As(err, &cliErr) {
se := StructuredError{
Error: cliErr.Msg,
Kind: cliErr.Kind,
Token: cliErr.Token,
Verb: cliErr.Verb,
}
switch cliErr.Kind {
case "unknown-verb":
se.Hint = "Run 'mlr -l' for a list of verbs, or 'mlr help verb <name>' for details."
case "unknown-flag":
se.Hint = "Run 'mlr --help' for a list of main flags."
case "verb-option-error":
if cliErr.Verb != "" {
se.Hint = "Run 'mlr " + cliErr.Verb + " --help' for a list of options."
}
}
return se
}
// Fallback: categorize by message pattern.
msg := err.Error()
// "mlr {verb}: option "{flag}" not recognized"
if strings.Contains(msg, ": option ") && strings.Contains(msg, "not recognized") {
verb := extractVerbFromMsg(msg)
token := extractQuotedToken(msg)
return StructuredError{
Error: msg,
Kind: "verb-option-error",
Token: token,
Verb: verb,
Hint: fmt.Sprintf("Run 'mlr %s --help' for a list of options.", verb),
}
}
// DSL parse errors
if strings.Contains(msg, "cannot parse DSL") || strings.Contains(msg, "DSL expression") {
return StructuredError{
Error: msg,
Kind: "dsl-parse-error",
Hint: "Run 'mlr put --help' for DSL syntax reference.",
}
}
return StructuredError{
Error: msg,
Kind: "generic",
}
}
// nearMatches populates did_you_mean for the structured error.
func nearMatches(se StructuredError) []string {
if se.Token == "" {
return nil
}
query := strings.ToLower(se.Token)
var candidates []string
switch se.Kind {
case "unknown-verb":
candidates = transformers.GetVerbNames()
case "unknown-flag":
candidates = cli.FLAG_TABLE.GetFlagNames()
case "verb-option-error":
// Use structured OptionSpec when available (from PR3 catalog).
if se.Verb != "" {
if info := transformers.GetVerbInfoForJSON(se.Verb); info != nil {
for _, opt := range info.Options {
candidates = append(candidates, opt.Flag)
}
}
}
if len(candidates) == 0 {
candidates = cli.FLAG_TABLE.GetFlagNames()
}
case "dsl-parse-error":
candidates = append(cst.BuiltinFunctionManagerInstance.GetBuiltinFunctionNames(),
cst.GetKeywordNames()...)
}
return topMatches(query, candidates, 3, levenshteinThreshold(query))
}
// topMatches returns up to n candidates from the list sorted by edit distance,
// keeping only those within maxDist of the query.
func topMatches(query string, candidates []string, n, maxDist int) []string {
type scored struct {
name string
dist int
}
var scored_ []scored
for _, c := range candidates {
d := levenshtein(query, strings.ToLower(c))
if d <= maxDist {
scored_ = append(scored_, scored{c, d})
}
}
sort.Slice(scored_, func(i, j int) bool {
if scored_[i].dist != scored_[j].dist {
return scored_[i].dist < scored_[j].dist
}
return scored_[i].name < scored_[j].name
})
result := make([]string, 0, n)
for i, s := range scored_ {
if i >= n {
break
}
result = append(result, s.name)
}
return result
}
// levenshteinThreshold returns the maximum edit distance to consider a match.
// Shorter queries use a tighter threshold.
func levenshteinThreshold(query string) int {
n := len(query)
switch {
case n <= 3:
return 1
case n <= 6:
return 2
default:
return 3
}
}
// ----------------------------------------------------------------
// Levenshtein edit distance (Wagner-Fischer, O(m*n) time, O(n) space)
func levenshtein(a, b string) int {
ra, rb := []rune(a), []rune(b)
la, lb := len(ra), len(rb)
if la == 0 {
return lb
}
if lb == 0 {
return la
}
prev := make([]int, lb+1)
curr := make([]int, lb+1)
for j := range prev {
prev[j] = j
}
for i, ca := range ra {
curr[0] = i + 1
for j, cb := range rb {
cost := 1
if ca == cb {
cost = 0
}
curr[j+1] = min3(curr[j]+1, prev[j+1]+1, prev[j]+cost)
}
prev, curr = curr, prev
}
return prev[lb]
}
func min3(a, b, c int) int {
if a < b {
if a < c {
return a
}
return c
}
if b < c {
return b
}
return c
}
// ----------------------------------------------------------------
// Helpers for string-based categorization
// extractTokenFromVerbError extracts the quoted token from a verb error
// such as `mlr cut: option "--foo" not recognized`.
// Used in mlrcli_parse.go to populate CLIError.Token from the error string.
func extractTokenFromVerbError(msg string) string {
return extractQuotedToken(msg)
}
// extractVerbFromMsg extracts the verb from "mlr {verb}: ..." error messages.
func extractVerbFromMsg(msg string) string {
// Pattern: "mlr {verb}: ..."
msg = strings.TrimPrefix(msg, "mlr ")
if i := strings.Index(msg, ":"); i > 0 {
return msg[:i]
}
return ""
}
// extractQuotedToken extracts the first double-quoted token from a message.
func extractQuotedToken(msg string) string {
start := strings.Index(msg, "\"")
if start < 0 {
return ""
}
end := strings.Index(msg[start+1:], "\"")
if end < 0 {
return ""
}
return msg[start+1 : start+1+end]
}

View file

@ -0,0 +1,177 @@
package climain
import (
"fmt"
"testing"
)
// ----------------------------------------------------------------
// Levenshtein
func TestLevenshteinIdentical(t *testing.T) {
if d := levenshtein("filter", "filter"); d != 0 {
t.Errorf("identical strings: got %d, want 0", d)
}
}
func TestLevenshteinEmpty(t *testing.T) {
if d := levenshtein("", "abc"); d != 3 {
t.Errorf("empty a: got %d, want 3", d)
}
if d := levenshtein("abc", ""); d != 3 {
t.Errorf("empty b: got %d, want 3", d)
}
}
func TestLevenshteinSubstitution(t *testing.T) {
// "fliter" → "filter": swap l↔i at positions 1-2 → distance 2
d := levenshtein("fliter", "filter")
if d != 2 {
t.Errorf("fliter/filter: got %d, want 2", d)
}
}
func TestLevenshteinInsertion(t *testing.T) {
// "--jsonn" → "--json": 1 deletion
if d := levenshtein("--jsonn", "--json"); d != 1 {
t.Errorf("--jsonn/--json: got %d, want 1", d)
}
}
func TestLevenshteinDeletion(t *testing.T) {
// "--complemment" → "--complement": 1 extra m
if d := levenshtein("--complemment", "--complement"); d != 1 {
t.Errorf("--complemment/--complement: got %d, want 1", d)
}
}
// ----------------------------------------------------------------
// topMatches
func TestTopMatchesBasic(t *testing.T) {
// "filtter" is distance 1 from "filter" (extra t), clearly the top match.
candidates := []string{"filter", "flatten", "fraction", "grep", "head"}
matches := topMatches("filtter", candidates, 3, 2)
if len(matches) == 0 {
t.Fatal("expected at least one match for 'filtter'")
}
if matches[0] != "filter" {
t.Errorf("expected 'filter' as top match, got %q", matches[0])
}
}
func TestTopMatchesNoneWithinThreshold(t *testing.T) {
candidates := []string{"aaaaa", "bbbbb", "ccccc"}
matches := topMatches("xyz", candidates, 3, 1)
if len(matches) != 0 {
t.Errorf("expected no matches, got %v", matches)
}
}
func TestTopMatchesRespectsCap(t *testing.T) {
// All candidates are within distance 1 of "a"; cap at 2
candidates := []string{"aa", "ab", "ac", "ad", "ae"}
matches := topMatches("a", candidates, 2, 2)
if len(matches) > 2 {
t.Errorf("expected at most 2 matches, got %d", len(matches))
}
}
// ----------------------------------------------------------------
// levenshteinThreshold
func TestThresholdShortQuery(t *testing.T) {
if levenshteinThreshold("ab") != 1 {
t.Error("2-char query should have threshold 1")
}
}
func TestThresholdMediumQuery(t *testing.T) {
if levenshteinThreshold("filter") != 2 {
t.Error("6-char query should have threshold 2")
}
}
func TestThresholdLongQuery(t *testing.T) {
if levenshteinThreshold("--complement") != 3 {
t.Error("12-char query should have threshold 3")
}
}
// ----------------------------------------------------------------
// WantErrorsJSON
func TestWantErrorsJSONFlag(t *testing.T) {
if !WantErrorsJSON([]string{"mlr", "--errors-json", "flitre"}) {
t.Error("should detect --errors-json")
}
if !WantErrorsJSON([]string{"mlr", "flitre", "--errors-json"}) {
t.Error("should detect --errors-json in any position")
}
if WantErrorsJSON([]string{"mlr", "flitre"}) {
t.Error("should not detect --errors-json when absent")
}
}
// ----------------------------------------------------------------
// CLIError round-trip
func TestCLIErrorInterface(t *testing.T) {
err := &CLIError{Kind: "unknown-verb", Token: "foo", Msg: "mlr: verb \"foo\" not found"}
if err.Error() != err.Msg {
t.Errorf("Error() should return Msg, got %q", err.Error())
}
}
// ----------------------------------------------------------------
// categorize
func TestCategorizeUnknownVerb(t *testing.T) {
err := &CLIError{Kind: "unknown-verb", Token: "flitre", Msg: "mlr: verb \"flitre\" not found"}
se := categorize(err)
if se.Kind != "unknown-verb" {
t.Errorf("kind: got %q, want unknown-verb", se.Kind)
}
if se.Token != "flitre" {
t.Errorf("token: got %q, want flitre", se.Token)
}
if se.Hint == "" {
t.Error("hint should be non-empty for unknown-verb")
}
}
func TestCategorizeUnknownFlag(t *testing.T) {
err := &CLIError{Kind: "unknown-flag", Token: "--jsonn", Msg: "mlr: option \"--jsonn\" not recognized"}
se := categorize(err)
if se.Kind != "unknown-flag" {
t.Errorf("kind: got %q, want unknown-flag", se.Kind)
}
if se.Hint == "" {
t.Error("hint should be non-empty for unknown-flag")
}
}
func TestCategorizeVerbOptionError(t *testing.T) {
err := &CLIError{
Kind: "verb-option-error", Token: "--bad", Verb: "cut",
Msg: "mlr cut: option \"--bad\" not recognized",
}
se := categorize(err)
if se.Kind != "verb-option-error" {
t.Errorf("kind: got %q, want verb-option-error", se.Kind)
}
if se.Verb != "cut" {
t.Errorf("verb: got %q, want cut", se.Verb)
}
if se.Hint == "" {
t.Error("hint should be non-empty for verb-option-error")
}
}
func TestCategorizeGenericFallback(t *testing.T) {
err := fmt.Errorf("some unexpected error")
se := categorize(err)
if se.Kind != "generic" {
t.Errorf("kind: got %q, want generic", se.Kind)
}
}

View file

@ -110,7 +110,10 @@ func ParseCommandLine(
args = lib.Getoptify(args)
// Pass one as described at the top of this file.
flagSequences, terminalSequence, verbSequences, dataFileNames := parseCommandLinePassOne(args)
flagSequences, terminalSequence, verbSequences, dataFileNames, passOneErr := parseCommandLinePassOne(args)
if passOneErr != nil {
return nil, nil, passOneErr
}
// Pass two as described at the top of this file.
return parseCommandLinePassTwo(flagSequences, terminalSequence, verbSequences, dataFileNames)
@ -124,6 +127,7 @@ func parseCommandLinePassOne(
terminalSequence []string,
verbSequences [][]string,
dataFileNames []string,
parseErr error,
) {
flagSequences = [][]string{}
terminalSequence = nil
@ -172,10 +176,14 @@ func parseCommandLinePassOne(
argi += 1
} else {
// Unrecognized main-flag. Fatal it here, and don't send it to pass two.
fmt.Fprintf(os.Stderr, "%s: option \"%s\" not recognized.\n", "mlr", args[argi])
fmt.Fprintf(os.Stderr, "Please run \"%s --help\" for usage information.\n", "mlr")
os.Exit(1)
// Unrecognized main-flag: return a structured error so the
// caller can emit JSON or prose depending on --errors-json.
parseErr = &CLIError{
Kind: "unknown-flag",
Token: args[argi],
Msg: fmt.Sprintf("mlr: option \"%s\" not recognized. Please run \"mlr --help\" for usage information.", args[argi]),
}
return
}
} else if onFirst && terminals.Dispatchable(args[argi]) {
@ -201,10 +209,12 @@ func parseCommandLinePassOne(
transformerSetup := transformers.LookUp(verb)
if transformerSetup == nil {
fmt.Fprintf(os.Stderr,
"mlr: verb \"%s\" not found. Please use \"mlr -l\" for a list.\n",
verb)
os.Exit(1)
parseErr = &CLIError{
Kind: "unknown-verb",
Token: verb,
Msg: fmt.Sprintf("mlr: verb \"%s\" not found. Please use \"mlr -l\" for a list.", verb),
}
return
}
// ParseCLIFunc returns (nil, nil) on pass-one success, (nil, err) on failure.
@ -222,8 +232,15 @@ func parseCommandLinePassOne(
if errors.Is(err, cli.ErrUsagePrinted) {
os.Exit(1)
}
fmt.Fprintf(os.Stderr, "mlr: %v\n", err)
os.Exit(1)
// Return a structured error so the caller can emit JSON or
// prose depending on --errors-json.
parseErr = &CLIError{
Kind: "verb-option-error",
Token: extractTokenFromVerbError(err.Error()),
Verb: verb,
Msg: err.Error(), // already "mlr {verb}: ..."
}
return
}
// For pass one we want the verbs to identify the arg-sequences
// they own within the command line, but not construct
@ -257,7 +274,7 @@ func parseCommandLinePassOne(
}
}
return flagSequences, terminalSequence, verbSequences, dataFileNames
return flagSequences, terminalSequence, verbSequences, dataFileNames, nil
}
// parseCommandLinePassTwo is as described at the top of this file.

View file

@ -40,9 +40,15 @@ func Main() MainReturn {
// found then this function will not return.
auxents.Dispatch(os.Args)
wantJSON := climain.WantErrorsJSON(os.Args)
options, recordTransformers, err := climain.ParseCommandLine(os.Args)
if err != nil {
printError(err)
if wantJSON {
climain.EmitStructuredError(err)
} else {
printError(err)
}
os.Exit(1)
}
@ -61,13 +67,15 @@ func Main() MainReturn {
}
}
// printError prints err to stderr. Errors that already start with "mlr " (e.g.
// "mlr stats1: ...") are printed as-is to avoid double-prefixing.
// printError prints err to stderr. Errors that already carry an "mlr" prefix
// (e.g. "mlr stats1: ..." or "mlr: verb not found") are printed as-is to
// avoid double-prefixing.
func printError(err error) {
if strings.HasPrefix(err.Error(), "mlr ") {
fmt.Fprintf(os.Stderr, "%v\n", err)
msg := err.Error()
if strings.HasPrefix(msg, "mlr") {
fmt.Fprintf(os.Stderr, "%v\n", msg)
} else {
fmt.Fprintf(os.Stderr, "mlr: %v\n", err)
fmt.Fprintf(os.Stderr, "mlr: %v\n", msg)
}
}