Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-17 16:38:54 +00:00
Code Issues Projects Releases Wiki Activity

Add mlr describe schema/shape introspection verb (#2098 PR6) (#2132)

Browse source 
One output record per input field: types seen with counts, occurrence
count, null count, cardinality, min/max, and -- for fields within the
-n/--max-values cap -- the complete distinct-value list in first-seen
order. `mlr --ojson describe` is the machine-readable form; nested
types/values flatten in tabular formats.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
John Kerl 2026-07-03 18:43:49 -04:00 • committed by GitHub
parent 7f60e7da57
commit d415ca7655
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
33 changed files with 932 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

34
docs/src/manpage.md
View file

@ -225,9 +225,9 @@ This is simply a copy of what you should see on running `man mlr` at a command p
1mVERB LIST0m
altkv bar bootstrap case cat check clean-whitespace count-distinct count
count-similar cut decimate fill-down fill-empty filter flatten format-values
fraction gap grep group-by group-like gsub having-fields head histogram
json-parse json-stringify join label latin1-to-utf8 least-frequent
count-similar cut decimate describe fill-down fill-empty filter flatten
format-values fraction gap grep group-by group-like gsub having-fields head
histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
merge-fields most-frequent nest nothing put regularize remove-empty-columns
rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
skip-trivial-records sort sort-within-records sparsify split ssub stats1
@ -1143,6 +1143,34 @@ This is simply a copy of what you should see on running `man mlr` at a command p
-n {n} Decimation factor (default 10).
-h|--help Show this message.
1mdescribe0m
Usage: mlr describe [options]
Shows a compact schema for the input data: field names, types, and value shape.
Emits one output record per input field, with types seen, counts, cardinality,
min/max, and (for low-cardinality fields) the complete set of distinct values.
Output fields, one record per input field:
field_name name of the input field
types map from type name (int, string, etc.) to occurrence count
count number of records in which the field appears
null_count count of field values either empty string or JSON null
distinct_count count of distinct values for the field
min, max minimum/maximum field value (works for strings as well as numbers)
values all distinct values, in order first seen -- only for fields
whose distinct_count is within the -n limit
Notes:
* Distinctness is computed on string representations -- so 4.1 and 4.10 are counted as distinct here.
* Use `mlr --ojson describe` for a machine-readable JSON document; in tabular
output formats the types map and values array are flattened.
* See also the summary verb, which reports summary statistics (mean, percentiles, etc.).
Options:
-n|--max-values {n} List a field's distinct values only if it has at most {n} of
them; 0 suppresses the values array entirely. Defaults to
20.
-h|--help Show this message.
1mfill-down0m
Usage: mlr fill-down [options]
If a given record has a missing value for a given field, fill that from

34
docs/src/manpage.txt
View file

@ -204,9 +204,9 @@
1mVERB LIST0m
altkv bar bootstrap case cat check clean-whitespace count-distinct count
count-similar cut decimate fill-down fill-empty filter flatten format-values
fraction gap grep group-by group-like gsub having-fields head histogram
json-parse json-stringify join label latin1-to-utf8 least-frequent
count-similar cut decimate describe fill-down fill-empty filter flatten
format-values fraction gap grep group-by group-like gsub having-fields head
histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
merge-fields most-frequent nest nothing put regularize remove-empty-columns
rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
skip-trivial-records sort sort-within-records sparsify split ssub stats1
@ -1122,6 +1122,34 @@
-n {n} Decimation factor (default 10).
-h|--help Show this message.
1mdescribe0m
Usage: mlr describe [options]
Shows a compact schema for the input data: field names, types, and value shape.
Emits one output record per input field, with types seen, counts, cardinality,
min/max, and (for low-cardinality fields) the complete set of distinct values.
Output fields, one record per input field:
field_name name of the input field
types map from type name (int, string, etc.) to occurrence count
count number of records in which the field appears
null_count count of field values either empty string or JSON null
distinct_count count of distinct values for the field
min, max minimum/maximum field value (works for strings as well as numbers)
values all distinct values, in order first seen -- only for fields
whose distinct_count is within the -n limit
Notes:
* Distinctness is computed on string representations -- so 4.1 and 4.10 are counted as distinct here.
* Use `mlr --ojson describe` for a machine-readable JSON document; in tabular
output formats the types map and values array are flattened.
* See also the summary verb, which reports summary statistics (mean, percentiles, etc.).
Options:
-n|--max-values {n} List a field's distinct values only if it has at most {n} of
them; 0 suppresses the values array entirely. Defaults to
20.
-h|--help Show this message.
1mfill-down0m
Usage: mlr fill-down [options]
If a given record has a missing value for a given field, fill that from

1
docs/src/reference-main-flag-list.md
View file

@ -299,6 +299,7 @@ These are flags which don't fit into any other category.
**Flags:**
* `--errors-json`: Emit parse errors as a JSON object to stderr instead of a plain text message. Intended for AI agents and scripts that branch on error kind rather than regex-matching prose. Equivalent to setting the `MLR_ERRORS_JSON` environment variable to a truthy value.
* `--fflush`: Force buffered output to be written after every output record. The default is flush output after every record if the output is to the terminal, or less often if the output is to a file or a pipe. The default is a significant performance optimization for large files. Use this flag to force frequent updates even when output is to a pipe or file, at a performance cost.
* `--files {filename}`: Use this to specify a file which itself contains, one per line, names of input files. May be used more than once.
* `--from {filename}`: Use this to specify an input file before the verb(s), rather than after. May be used more than once. Example: `mlr --from a.dat --from b.dat cat` is the same as `mlr cat a.dat b.dat`.

126
docs/src/reference-verbs.md
View file

@ -890,6 +890,132 @@ Options:
-h|--help Show this message.
</pre>
## describe
<pre class="pre-highlight-in-pair">
<b>mlr describe --help</b>
</pre>
<pre class="pre-non-highlight-in-pair">
Usage: mlr describe [options]
Shows a compact schema for the input data: field names, types, and value shape.
Emits one output record per input field, with types seen, counts, cardinality,
min/max, and (for low-cardinality fields) the complete set of distinct values.
Output fields, one record per input field:
field_name name of the input field
types map from type name (int, string, etc.) to occurrence count
count number of records in which the field appears
null_count count of field values either empty string or JSON null
distinct_count count of distinct values for the field
min, max minimum/maximum field value (works for strings as well as numbers)
values all distinct values, in order first seen -- only for fields
whose distinct_count is within the -n limit
Notes:
* Distinctness is computed on string representations -- so 4.1 and 4.10 are counted as distinct here.
* Use `mlr --ojson describe` for a machine-readable JSON document; in tabular
output formats the types map and values array are flattened.
* See also the summary verb, which reports summary statistics (mean, percentiles, etc.).
Options:
-n|--max-values {n} List a field's distinct values only if it has at most {n} of
them; 0 suppresses the values array entirely. Defaults to
20.
-h|--help Show this message.
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr --icsv --ojson describe example.csv</b>
</pre>
<pre class="pre-non-highlight-in-pair">
[
{
"field_name": "color",
"types": {
"string": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 3,
"min": "purple",
"max": "yellow",
"values": ["yellow", "red", "purple"]
},
{
"field_name": "shape",
"types": {
"string": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 3,
"min": "circle",
"max": "triangle",
"values": ["triangle", "square", "circle"]
},
{
"field_name": "flag",
"types": {
"string": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 2,
"min": "false",
"max": "true",
"values": ["true", "false"]
},
{
"field_name": "k",
"types": {
"int": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 10,
"min": 1,
"max": 10,
"values": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
},
{
"field_name": "index",
"types": {
"int": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 10,
"min": 11,
"max": 91,
"values": [11, 15, 16, 48, 51, 64, 65, 73, 87, 91]
},
{
"field_name": "quantity",
"types": {
"float": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 10,
"min": 13.8103,
"max": 81.229,
"values": [43.6498, 79.2778, 13.8103, 77.5542, 81.2290, 77.1991, 80.1405, 63.9785, 63.5058, 72.3735]
},
{
"field_name": "rate",
"types": {
"float": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 10,
"min": 0.013,
"max": 9.887,
"values": [9.8870, 0.0130, 2.9010, 7.4670, 8.5910, 9.5310, 5.8240, 4.2370, 8.3350, 8.2430]
}
]
</pre>
## fill-down
<pre class="pre-highlight-in-pair">

10
docs/src/reference-verbs.md.in
View file

@ -329,6 +329,16 @@ GENMD-RUN-COMMAND
mlr decimate --help
GENMD-EOF
## describe
GENMD-RUN-COMMAND
mlr describe --help
GENMD-EOF
GENMD-RUN-COMMAND
mlr --icsv --ojson describe example.csv
GENMD-EOF
## fill-down
GENMD-RUN-COMMAND

34
man/manpage.txt
View file

@ -204,9 +204,9 @@
1mVERB LIST0m
altkv bar bootstrap case cat check clean-whitespace count-distinct count
count-similar cut decimate fill-down fill-empty filter flatten format-values
fraction gap grep group-by group-like gsub having-fields head histogram
json-parse json-stringify join label latin1-to-utf8 least-frequent
count-similar cut decimate describe fill-down fill-empty filter flatten
format-values fraction gap grep group-by group-like gsub having-fields head
histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
merge-fields most-frequent nest nothing put regularize remove-empty-columns
rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
skip-trivial-records sort sort-within-records sparsify split ssub stats1
@ -1122,6 +1122,34 @@
-n {n} Decimation factor (default 10).
-h|--help Show this message.
1mdescribe0m
Usage: mlr describe [options]
Shows a compact schema for the input data: field names, types, and value shape.
Emits one output record per input field, with types seen, counts, cardinality,
min/max, and (for low-cardinality fields) the complete set of distinct values.
Output fields, one record per input field:
field_name name of the input field
types map from type name (int, string, etc.) to occurrence count
count number of records in which the field appears
null_count count of field values either empty string or JSON null
distinct_count count of distinct values for the field
min, max minimum/maximum field value (works for strings as well as numbers)
values all distinct values, in order first seen -- only for fields
whose distinct_count is within the -n limit
Notes:
* Distinctness is computed on string representations -- so 4.1 and 4.10 are counted as distinct here.
* Use `mlr --ojson describe` for a machine-readable JSON document; in tabular
output formats the types map and values array are flattened.
* See also the summary verb, which reports summary statistics (mean, percentiles, etc.).
Options:
-n|--max-values {n} List a field's distinct values only if it has at most {n} of
them; 0 suppresses the values array entirely. Defaults to
20.
-h|--help Show this message.
1mfill-down0m
Usage: mlr fill-down [options]
If a given record has a missing value for a given field, fill that from

40
man/mlr.1
View file

@ -246,9 +246,9 @@ for all things with "map" in their names.
.\}
.nf
altkv bar bootstrap case cat check clean-whitespace count-distinct count
count-similar cut decimate fill-down fill-empty filter flatten format-values
fraction gap grep group-by group-like gsub having-fields head histogram
json-parse json-stringify join label latin1-to-utf8 least-frequent
count-similar cut decimate describe fill-down fill-empty filter flatten
format-values fraction gap grep group-by group-like gsub having-fields head
histogram json-parse json-stringify join label latin1-to-utf8 least-frequent
merge-fields most-frequent nest nothing put regularize remove-empty-columns
rename reorder repeat reshape sample sec2gmtdate sec2gmt seqgen shuffle
skip-trivial-records sort sort-within-records sparsify split ssub stats1
@ -1384,6 +1384,40 @@ Options:
.fi
.if n \{\
.RE
.SS "describe"
.if n \{\
.RS 0
.\}
.nf
Usage: mlr describe [options]
Shows a compact schema for the input data: field names, types, and value shape.
Emits one output record per input field, with types seen, counts, cardinality,
min/max, and (for low-cardinality fields) the complete set of distinct values.
Output fields, one record per input field:
field_name name of the input field
types map from type name (int, string, etc.) to occurrence count
count number of records in which the field appears
null_count count of field values either empty string or JSON null
distinct_count count of distinct values for the field
min, max minimum/maximum field value (works for strings as well as numbers)
values all distinct values, in order first seen -- only for fields
whose distinct_count is within the -n limit
Notes:
* Distinctness is computed on string representations -- so 4.1 and 4.10 are counted as distinct here.
* Use `mlr --ojson describe` for a machine-readable JSON document; in tabular
output formats the types map and values array are flattened.
* See also the summary verb, which reports summary statistics (mean, percentiles, etc.).
Options:
-n|--max-values {n} List a field's distinct values only if it has at most {n} of
them; 0 suppresses the values array entirely. Defaults to
20.
-h|--help Show this message.
.fi
.if n \{\
.RE
.SS "fill-down"
.if n \{\
.RS 0

1
pkg/transformers/aaa_transformer_table.go
View file

@ -22,6 +22,7 @@ var TRANSFORMER_LOOKUP_TABLE = []TransformerSetup{
CountSimilarSetup,
CutSetup,
DecimateSetup,
DescribeSetup,
FillDownSetup,
FillEmptySetup,
FilterSetup,

255
pkg/transformers/describe.go Normal file
View file

@ -0,0 +1,255 @@
// The describe verb reports a compact schema/shape for the input data: one
// output record per input field, with the types seen, occurrence counts,
// cardinality, min/max, and -- for low-cardinality fields -- the complete set
// of distinct values.
//
// This is PR6 of the AI-friendly roadmap (issue #2098): `mlr summary` is the
// statistician's view (means, percentiles, outlier fences); `describe` is the
// agent's view -- just enough shape for a person or an LLM agent to construct
// a correct next command. In particular the distinct-value list for
// low-cardinality fields gives the actual domain for flags like `-g` and for
// DSL comparisons, attacking value-hallucination with data-derived values the
// static help catalog cannot know.
//
// Use `mlr --ojson describe ...` for a machine-readable JSON document: the
// per-field `types` map and `values` array nest naturally there, and flatten
// in tabular output formats.
package transformers
import (
"fmt"
"os"
"strings"
"github.com/johnkerl/miller/v6/pkg/cli"
"github.com/johnkerl/miller/v6/pkg/lib"
"github.com/johnkerl/miller/v6/pkg/mlrval"
"github.com/johnkerl/miller/v6/pkg/transformers/utils"
"github.com/johnkerl/miller/v6/pkg/types"
)
const verbNameDescribe = "describe"
const describeDefaultMaxValues = 20
var describeOptions = []OptionSpec{
{Flag: "-n", Aliases: []string{"--max-values"}, Arg: "{n}", Type: "int",
Desc: fmt.Sprintf("List a field's distinct values only if it has at most {n} of them; 0 suppresses the values array entirely. Defaults to %d.", describeDefaultMaxValues)},
}
var DescribeSetup = TransformerSetup{
Verb: verbNameDescribe,
UsageFunc: transformerDescribeUsage,
ParseCLIFunc: transformerDescribeParseCLI,
IgnoresInput: false,
Options: describeOptions,
}
func transformerDescribeUsage(
o *os.File,
) {
fmt.Fprintf(o, "Usage: %s %s [options]\n", "mlr", verbNameDescribe)
fmt.Fprintf(o, "Shows a compact schema for the input data: field names, types, and value shape.\n")
fmt.Fprintf(o, "Emits one output record per input field, with types seen, counts, cardinality,\n")
fmt.Fprintf(o, "min/max, and (for low-cardinality fields) the complete set of distinct values.\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Output fields, one record per input field:\n")
fmt.Fprintf(o, " field_name name of the input field\n")
fmt.Fprintf(o, " types map from type name (int, string, etc.) to occurrence count\n")
fmt.Fprintf(o, " count number of records in which the field appears\n")
fmt.Fprintf(o, " null_count count of field values either empty string or JSON null\n")
fmt.Fprintf(o, " distinct_count count of distinct values for the field\n")
fmt.Fprintf(o, " min, max minimum/maximum field value (works for strings as well as numbers)\n")
fmt.Fprintf(o, " values all distinct values, in order first seen -- only for fields\n")
fmt.Fprintf(o, " whose distinct_count is within the -n limit\n")
fmt.Fprintf(o, "\n")
fmt.Fprintf(o, "Notes:\n")
fmt.Fprintf(o, "* Distinctness is computed on string representations -- so 4.1 and 4.10 are counted as distinct here.\n")
fmt.Fprintf(o, "* Use `mlr --ojson describe` for a machine-readable JSON document; in tabular\n")
fmt.Fprintf(o, " output formats the types map and values array are flattened.\n")
fmt.Fprintf(o, "* See also the summary verb, which reports summary statistics (mean, percentiles, etc.).\n")
fmt.Fprintf(o, "\n")
WriteVerbOptions(o, describeOptions)
}
func transformerDescribeParseCLI(
pargi *int,
argc int,
args []string,
_ *cli.TOptions,
doConstruct bool, // false for first pass of CLI-parse, true for second pass
) (RecordTransformer, error) {
// Skip the verb name from the current spot in the mlr command line
argi := *pargi
verb := args[argi]
argi++
maxValues := int64(describeDefaultMaxValues)
for argi < argc /* variable increment: 1 or 2 depending on flag */ {
opt := args[argi]
if !strings.HasPrefix(opt, "-") {
break // No more flag options to process
}
if args[argi] == "--" {
break // All transformers must do this so main-flags can follow verb-flags
}
argi++
switch opt {
case "-h", "--help":
transformerDescribeUsage(os.Stdout)
return nil, cli.ErrHelpRequested
case "-n", "--max-values":
n, err := cli.VerbGetIntArg(verb, opt, args, &argi, argc)
if err != nil {
return nil, err
}
if n < 0 {
return nil, cli.VerbErrorf(verb, "option \"%s\" requires a non-negative value", opt)
}
maxValues = n
default:
return nil, cli.VerbErrorf(verb, "option \"%s\" not recognized", opt)
}
}
*pargi = argi
if !doConstruct { // All transformers must do this for main command-line parsing
return nil, nil
}
return NewTransformerDescribe(maxValues)
}
// tFieldDescription accumulates per-field shape information across the record
// stream.
type tFieldDescription struct {
// Needs lib.OrderedMap, not map[string]int64, for deterministic output.
// Type name -> occurrence count; a map in case of heterogeneous data.
typesMap *lib.OrderedMap[int64]
count int64
nullCount int64
// String representation -> first-encountered value, in order first seen.
// Distinctness is on string representations, matching the summary verb's
// distinct_count; keeping the original mlrval preserves int/float/etc.
// types in the emitted values array.
distinctValues *lib.OrderedMap[*mlrval.Mlrval]
minAccumulator utils.IStats1Accumulator
maxAccumulator utils.IStats1Accumulator
}
func newFieldDescription() *tFieldDescription {
return &tFieldDescription{
typesMap: lib.NewOrderedMap[int64](),
distinctValues: lib.NewOrderedMap[*mlrval.Mlrval](),
minAccumulator: utils.NewStats1MinAccumulator(),
maxAccumulator: utils.NewStats1MaxAccumulator(),
}
}
type TransformerDescribe struct {
fieldDescriptions *lib.OrderedMap[*tFieldDescription]
maxValues int64
}
func NewTransformerDescribe(
maxValues int64,
) (*TransformerDescribe, error) {
return &TransformerDescribe{
fieldDescriptions: lib.NewOrderedMap[*tFieldDescription](),
maxValues: maxValues,
}, nil
}
func (tr *TransformerDescribe) Transform(
inrecAndContext *types.RecordAndContext,
outputRecordsAndContexts *[]*types.RecordAndContext, // list of *types.RecordAndContext
inputDownstreamDoneChannel <-chan bool,
outputDownstreamDoneChannel chan<- bool,
) {
HandleDefaultDownstreamDone(inputDownstreamDoneChannel, outputDownstreamDoneChannel)
if !inrecAndContext.EndOfStream {
tr.ingest(inrecAndContext)
} else {
tr.emit(inrecAndContext, outputRecordsAndContexts)
}
}
func (tr *TransformerDescribe) ingest(
inrecAndContext *types.RecordAndContext,
) {
inrec := inrecAndContext.Record
for pe := inrec.Head; pe != nil; pe = pe.Next {
fieldDescription := tr.fieldDescriptions.Get(pe.Key)
if fieldDescription == nil {
fieldDescription = newFieldDescription()
tr.fieldDescriptions.Put(pe.Key, fieldDescription)
}
typeName := pe.Value.GetTypeName()
typeCount, ok := fieldDescription.typesMap.GetWithCheck(typeName)
if !ok {
fieldDescription.typesMap.Put(typeName, int64(1))
} else {
fieldDescription.typesMap.Put(typeName, typeCount+1)
}
fieldDescription.count++
if pe.Value.IsVoid() || pe.Value.IsNull() {
fieldDescription.nullCount++
}
valueString := pe.Value.String()
if !fieldDescription.distinctValues.Has(valueString) {
fieldDescription.distinctValues.Put(valueString, pe.Value.Copy())
}
fieldDescription.minAccumulator.Ingest(pe.Value)
fieldDescription.maxAccumulator.Ingest(pe.Value)
}
}
func (tr *TransformerDescribe) emit(
inrecAndContext *types.RecordAndContext,
outputRecordsAndContexts *[]*types.RecordAndContext, // list of *types.RecordAndContext
) {
for pe := tr.fieldDescriptions.Head; pe != nil; pe = pe.Next {
fieldDescription := pe.Value
newrec := mlrval.NewMlrmapAsRecord()
newrec.PutCopy("field_name", mlrval.FromString(pe.Key))
typesMap := mlrval.NewMlrmap()
for pf := fieldDescription.typesMap.Head; pf != nil; pf = pf.Next {
typesMap.PutReference(pf.Key, mlrval.FromInt(pf.Value))
}
newrec.PutReference("types", mlrval.FromMap(typesMap))
newrec.PutReference("count", mlrval.FromInt(fieldDescription.count))
newrec.PutReference("null_count", mlrval.FromInt(fieldDescription.nullCount))
newrec.PutReference("distinct_count", mlrval.FromInt(fieldDescription.distinctValues.FieldCount))
newrec.PutCopy("min", fieldDescription.minAccumulator.Emit())
newrec.PutCopy("max", fieldDescription.maxAccumulator.Emit())
if tr.maxValues > 0 && fieldDescription.distinctValues.FieldCount <= tr.maxValues {
values := make([]*mlrval.Mlrval, 0, fieldDescription.distinctValues.FieldCount)
for pf := fieldDescription.distinctValues.Head; pf != nil; pf = pf.Next {
values = append(values, pf.Value)
}
newrec.PutReference("values", mlrval.FromArray(values))
}
*outputRecordsAndContexts = append(*outputRecordsAndContexts, types.NewRecordAndContext(newrec, &inrecAndContext.Context))
}
*outputRecordsAndContexts = append(*outputRecordsAndContexts, inrecAndContext) // end-of-stream marker
}

21
plans/plan-2098-llm.md
View file

@ -251,7 +251,7 @@ validation path.
---
## PR 6 — `mlr describe` schema/shape introspection
## PR 6 — `mlr describe` schema/shape introspection *(landed)*
**Goal.** Let an agent learn the *data's* shape, complementing the catalog's
*tool* shape.
@ -261,6 +261,25 @@ validation path.
- Leverage Miller's existing type-inference (`pkg/mlrval`) and field-collection
machinery; likely a new verb in `pkg/transformers/`.
**Landed.** New verb `describe` (`pkg/transformers/describe.go`), registered in
`TRANSFORMER_LOOKUP_TABLE` with Tier-2 `Options` so it appears structured in
the PR1 catalog and PR2 index automatically. One output record per input
field: `field_name`, `types` (type-name → occurrence-count map, via
`GetTypeName` type inference), `count`, `null_count`, `distinct_count`,
`min`/`max`, and — for fields whose cardinality is within `-n`/`--max-values`
(default 20; 0 suppresses) — a `values` array listing every distinct value in
first-seen order. The `values` list is the data-derived *constraint* domain
deferred out of PR3: an agent copies real values for `-g`, DSL comparisons,
etc. instead of guessing. The JSON form is Miller-native — `mlr --ojson
describe` — with `types`/`values` nesting in JSON and auto-flattening in
tabular formats, so no verb-level `--as-json` flag was needed; `describe` is
positioned relative to `summary` as schema-shape vs. summary-statistics.
Distinctness is on string representations, matching `summary`'s
`distinct_count`; null semantics (empty or JSON null) match `summary`'s
`null_count`. Tests: `test/cases/verb-describe/` (JSON, pprint-flattened,
heterogeneous input, `-n` cap, `-n 0`, null-vs-empty, bad-option); docs:
`## describe` in `reference-verbs.md.in`.
---
## PR 7 — MCP server + Agent Skill (the loop)

29
test/cases/cli-help/0001/expout
View file

@ -160,6 +160,35 @@ Options:
-n {n} Decimation factor (default 10).
-h|--help Show this message.
================================================================
describe
Usage: mlr describe [options]
Shows a compact schema for the input data: field names, types, and value shape.
Emits one output record per input field, with types seen, counts, cardinality,
min/max, and (for low-cardinality fields) the complete set of distinct values.
Output fields, one record per input field:
field_name name of the input field
types map from type name (int, string, etc.) to occurrence count
count number of records in which the field appears
null_count count of field values either empty string or JSON null
distinct_count count of distinct values for the field
min, max minimum/maximum field value (works for strings as well as numbers)
values all distinct values, in order first seen -- only for fields
whose distinct_count is within the -n limit
Notes:
* Distinctness is computed on string representations -- so 4.1 and 4.10 are counted as distinct here.
* Use `mlr --ojson describe` for a machine-readable JSON document; in tabular
output formats the types map and values array are flattened.
* See also the summary verb, which reports summary statistics (mean, percentiles, etc.).
Options:
-n|--max-values {n} List a field's distinct values only if it has at most {n} of
them; 0 suppresses the values array entirely. Defaults to
20.
-h|--help Show this message.
================================================================
fill-down
Usage: mlr fill-down [options]

1
test/cases/verb-describe/bad-option/cmd Normal file
View file

@ -0,0 +1 @@
mlr describe -q test/input/abixy

1
test/cases/verb-describe/bad-option/experr Normal file
View file

@ -0,0 +1 @@
mlr describe: option "-q" not recognized

0
test/cases/verb-describe/bad-option/expout Normal file
View file

0
test/cases/verb-describe/bad-option/should-fail Normal file
View file

1
test/cases/verb-describe/het/cmd Normal file
View file

@ -0,0 +1 @@
mlr --ojson describe test/input/abixy-het

0
test/cases/verb-describe/het/experr Normal file
View file

122
test/cases/verb-describe/het/expout Normal file
View file

@ -0,0 +1,122 @@
[
{
"field_name": "a",
"types": {
"string": 8
},
"count": 8,
"null_count": 0,
"distinct_count": 4,
"min": "eks",
"max": "zee",
"values": ["pan", "eks", "wye", "zee"]
},
{
"field_name": "b",
"types": {
"string": 8
},
"count": 8,
"null_count": 0,
"distinct_count": 3,
"min": "pan",
"max": "zee",
"values": ["pan", "wye", "zee"]
},
{
"field_name": "i",
"types": {
"int": 9
},
"count": 9,
"null_count": 0,
"distinct_count": 9,
"min": 1,
"max": 10,
"values": [1, 2, 3, 4, 5, 6, 8, 9, 10]
},
{
"field_name": "x",
"types": {
"float": 9
},
"count": 9,
"null_count": 0,
"distinct_count": 9,
"min": 0.03144188,
"max": 0.75867996,
"values": [0.34679014, 0.75867996, 0.20460331, 0.38139939, 0.52712616, 0.61178406, 0.59855401, 0.03144188, 0.50262601]
},
{
"field_name": "y",
"types": {
"float": 9
},
"count": 9,
"null_count": 0,
"distinct_count": 9,
"min": 0.13418874,
"max": 0.95261836,
"values": [0.72680286, 0.52215111, 0.33831853, 0.13418874, 0.86362447, 0.49322129, 0.18788492, 0.74955076, 0.95261836]
},
{
"field_name": "aaa",
"types": {
"string": 2
},
"count": 2,
"null_count": 0,
"distinct_count": 2,
"min": "hat",
"max": "wye",
"values": ["wye", "hat"]
},
{
"field_name": "bbb",
"types": {
"string": 2
},
"count": 2,
"null_count": 0,
"distinct_count": 1,
"min": "wye",
"max": "wye",
"values": ["wye"]
},
{
"field_name": "xxx",
"types": {
"float": 1
},
"count": 1,
"null_count": 0,
"distinct_count": 1,
"min": 0.57328892,
"max": 0.57328892,
"values": [0.57328892]
},
{
"field_name": "iii",
"types": {
"int": 1
},
"count": 1,
"null_count": 0,
"distinct_count": 1,
"min": 7,
"max": 7,
"values": [7]
},
{
"field_name": "yyy",
"types": {
"float": 1
},
"count": 1,
"null_count": 0,
"distinct_count": 1,
"min": 0.97618139,
"max": 0.97618139,
"values": [0.97618139]
}
]

1
test/cases/verb-describe/json/cmd Normal file
View file

@ -0,0 +1 @@
mlr --ojson describe test/input/abixy

0
test/cases/verb-describe/json/experr Normal file
View file

62
test/cases/verb-describe/json/expout Normal file
View file

@ -0,0 +1,62 @@
[
{
"field_name": "a",
"types": {
"string": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 5,
"min": "eks",
"max": "zee",
"values": ["pan", "eks", "wye", "zee", "hat"]
},
{
"field_name": "b",
"types": {
"string": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 3,
"min": "pan",
"max": "zee",
"values": ["pan", "wye", "zee"]
},
{
"field_name": "i",
"types": {
"int": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 10,
"min": 1,
"max": 10,
"values": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
},
{
"field_name": "x",
"types": {
"float": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 10,
"min": 0.03144188,
"max": 0.75867996,
"values": [0.34679014, 0.75867996, 0.20460331, 0.38139939, 0.57328892, 0.52712616, 0.61178406, 0.59855401, 0.03144188, 0.50262601]
},
{
"field_name": "y",
"types": {
"float": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 10,
"min": 0.13418874,
"max": 0.97618139,
"values": [0.72680286, 0.52215111, 0.33831853, 0.13418874, 0.86362447, 0.49322129, 0.18788492, 0.97618139, 0.74955076, 0.95261836]
}
]

1
test/cases/verb-describe/max-values/cmd Normal file
View file

@ -0,0 +1 @@
mlr --ojson describe -n 3 test/input/abixy

0
test/cases/verb-describe/max-values/experr Normal file
View file

58
test/cases/verb-describe/max-values/expout Normal file
View file

@ -0,0 +1,58 @@
[
{
"field_name": "a",
"types": {
"string": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 5,
"min": "eks",
"max": "zee"
},
{
"field_name": "b",
"types": {
"string": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 3,
"min": "pan",
"max": "zee",
"values": ["pan", "wye", "zee"]
},
{
"field_name": "i",
"types": {
"int": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 10,
"min": 1,
"max": 10
},
{
"field_name": "x",
"types": {
"float": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 10,
"min": 0.03144188,
"max": 0.75867996
},
{
"field_name": "y",
"types": {
"float": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 10,
"min": 0.13418874,
"max": 0.97618139
}
]

1
test/cases/verb-describe/no-values/cmd Normal file
View file

@ -0,0 +1 @@
mlr --ojson describe --max-values 0 test/input/abixy

0
test/cases/verb-describe/no-values/experr Normal file
View file

57
test/cases/verb-describe/no-values/expout Normal file
View file

@ -0,0 +1,57 @@
[
{
"field_name": "a",
"types": {
"string": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 5,
"min": "eks",
"max": "zee"
},
{
"field_name": "b",
"types": {
"string": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 3,
"min": "pan",
"max": "zee"
},
{
"field_name": "i",
"types": {
"int": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 10,
"min": 1,
"max": 10
},
{
"field_name": "x",
"types": {
"float": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 10,
"min": 0.03144188,
"max": 0.75867996
},
{
"field_name": "y",
"types": {
"float": 10
},
"count": 10,
"null_count": 0,
"distinct_count": 10,
"min": 0.13418874,
"max": 0.97618139
}
]

1
test/cases/verb-describe/null-vs-empty/cmd Normal file
View file

@ -0,0 +1 @@
mlr --ojson describe test/input/null-vs-empty.dkvp

0
test/cases/verb-describe/null-vs-empty/experr Normal file
View file

41
test/cases/verb-describe/null-vs-empty/expout Normal file
View file

@ -0,0 +1,41 @@
[
{
"field_name": "x",
"types": {
"int": 2,
"empty": 1
},
"count": 3,
"null_count": 1,
"distinct_count": 2,
"min": 1,
"max": "",
"values": [1, ""]
},
{
"field_name": "y",
"types": {
"int": 1,
"empty": 2
},
"count": 3,
"null_count": 2,
"distinct_count": 2,
"min": 2,
"max": "",
"values": [2, ""]
},
{
"field_name": "s",
"types": {
"string": 2,
"empty": 1
},
"count": 3,
"null_count": 1,
"distinct_count": 3,
"min": "",
"max": "hurrah",
"values": ["hello", "", "hurrah"]
}
]

1
test/cases/verb-describe/pprint/cmd Normal file
View file

@ -0,0 +1 @@
mlr --opprint describe test/input/abixy

0
test/cases/verb-describe/pprint/experr Normal file
View file

12
test/cases/verb-describe/pprint/expout Normal file
View file

@ -0,0 +1,12 @@
field_name types.string count null_count distinct_count min max values.1 values.2 values.3 values.4 values.5
a 10 10 0 5 eks zee pan eks wye zee hat
field_name types.string count null_count distinct_count min max values.1 values.2 values.3
b 10 10 0 3 pan zee pan wye zee
field_name types.int count null_count distinct_count min max values.1 values.2 values.3 values.4 values.5 values.6 values.7 values.8 values.9 values.10
i 10 10 0 10 1 10 1 2 3 4 5 6 7 8 9 10
field_name types.float count null_count distinct_count min max values.1 values.2 values.3 values.4 values.5 values.6 values.7 values.8 values.9 values.10
x 10 10 0 10 0.03144188 0.75867996 0.34679014 0.75867996 0.20460331 0.38139939 0.57328892 0.52712616 0.61178406 0.59855401 0.03144188 0.50262601
y 10 10 0 10 0.13418874 0.97618139 0.72680286 0.52215111 0.33831853 0.13418874 0.86362447 0.49322129 0.18788492 0.97618139 0.74955076 0.95261836