Add DCF (Debian Control Format) as supported file type (#1970)

* reader

* record writer

* pkg/output/record_writer_dcf.go

* test/input/test.dcf

* test/cases/io-dcf/

* pkg/cli/option_parse.go

* make fmt

* make dev

* docs and `make dev`
This commit is contained in:
John Kerl 2026-02-16 11:10:39 -05:00 committed by GitHub
parent a5f4c9d895
commit daae7ff7f0
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
44 changed files with 579 additions and 29 deletions

9
docs/src/data/sample.dcf Normal file
View file

@ -0,0 +1,9 @@
Package: foo
Version: 1.0
Depends: libc6 (>= 2.0), libfoo (>= 1.2)
Description: A test package.
Package: bar
Version: 2.0
Recommends: foo
Description: Another package.

View file

@ -17,7 +17,7 @@ Quick links:
# Features
Miller is like awk, sed, cut, join, and sort for **name-indexed data, such as
CSV, TSV, JSON, JSON Lines, and YAML**. You get to work with your data using named
CSV, TSV, JSON, JSON Lines, YAML, and DCF**. You get to work with your data using named
fields, without needing to count positional column indices.
This is something the Unix toolkit always could have done, and arguably always
@ -25,7 +25,7 @@ should have done. It operates on key-value-pair data while the familiar Unix
tools operate on integer-indexed fields: if the natural data structure for the
latter is the array, then Miller's natural data structure is the
insertion-ordered hash map. This encompasses a **variety of data formats**,
including but not limited to the familiar CSV, TSV, JSON, JSON Lines, and YAML.
including but not limited to the familiar CSV, TSV, JSON, JSON Lines, YAML, and DCF.
(Miller can handle **positionally-indexed data** as a special case.)
* Miller is **multi-purpose**: it's useful for **data cleaning**, **data reduction**, **statistical reporting**, **devops**, **system administration**, **log-file processing**, **format conversion**, and **database-query post-processing**.

View file

@ -1,7 +1,7 @@
# Features
Miller is like awk, sed, cut, join, and sort for **name-indexed data, such as
CSV, TSV, JSON, JSON Lines, and YAML**. You get to work with your data using named
CSV, TSV, JSON, JSON Lines, YAML, and DCF**. You get to work with your data using named
fields, without needing to count positional column indices.
This is something the Unix toolkit always could have done, and arguably always
@ -9,7 +9,7 @@ should have done. It operates on key-value-pair data while the familiar Unix
tools operate on integer-indexed fields: if the natural data structure for the
latter is the array, then Miller's natural data structure is the
insertion-ordered hash map. This encompasses a **variety of data formats**,
including but not limited to the familiar CSV, TSV, JSON, JSON Lines, and YAML.
including but not limited to the familiar CSV, TSV, JSON, JSON Lines, YAML, and DCF.
(Miller can handle **positionally-indexed data** as a special case.)
* Miller is **multi-purpose**: it's useful for **data cleaning**, **data reduction**, **statistical reporting**, **devops**, **system administration**, **log-file processing**, **format conversion**, and **database-query post-processing**.

View file

@ -17,7 +17,7 @@ Quick links:
# File formats
Miller handles name-indexed data using several formats: some you probably know
by name, such as CSV, TSV, JSON, JSON Lines, and YAML -- and other formats you're likely already
by name, such as CSV, TSV, JSON, JSON Lines, YAML, and DCF -- and other formats you're likely already
seeing and using in your structured data.
Additionally, Miller gives you the option to include comments within your data.
@ -108,6 +108,17 @@ NIDX: implicitly numerically indexed (Unix-toolkit style)
| the quick brown | Record 1: "1":"the", "2":"quick", "3":"brown"
| fox jumped | Record 2: "1":"fox", "2":"jumped"
+---------------------+
DCF: Debian control file format
+------------+
| apple: 1 |
| bat: 2 |
| cog: 3 |
| |
| dish: 7 |
| egg: 8 |
| 3: flint |
+------------+
</pre>
## CSV/TSV/ASV/USV/etc.
@ -720,6 +731,43 @@ the dawn's
light
</pre>
## DCF (Debian control file)
<pre class="pre-highlight-in-pair">
<b>cat data/sample.dcf</b>
</pre>
<pre class="pre-non-highlight-in-pair">
Package: foo
Version: 1.0
Depends: libc6 (>= 2.0), libfoo (>= 1.2)
Description: A test package.
Package: bar
Version: 2.0
Recommends: foo
Description: Another package.
</pre>
<pre class="pre-highlight-in-pair">
<b>mlr -i dcf -o json cat data/sample.dcf</b>
</pre>
<pre class="pre-non-highlight-in-pair">
[
{
"Package": "foo",
"Version": "1.0",
"Depends": ["libc6 (>= 2.0)", "libfoo (>= 1.2)"],
"Description": "A test package."
},
{
"Package": "bar",
"Version": "2.0",
"Recommends": ["foo"],
"Description": "Another package."
}
]
</pre>
## Data-conversion keystroke-savers
While you can do format conversion using `mlr --icsv --ojson cat myfile.csv`, there are also keystroke-savers for this purpose, such as `mlr --c2j cat myfile.csv`. For a complete list:
@ -731,7 +779,7 @@ While you can do format conversion using `mlr --icsv --ojson cat myfile.csv`, th
FORMAT-CONVERSION KEYSTROKE-SAVER FLAGS
As keystroke-savers for format-conversion you may use the following.
The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines,
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively.
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out).
| In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML |
+----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+

View file

@ -1,7 +1,7 @@
# File formats
Miller handles name-indexed data using several formats: some you probably know
by name, such as CSV, TSV, JSON, JSON Lines, and YAML -- and other formats you're likely already
by name, such as CSV, TSV, JSON, JSON Lines, YAML, and DCF -- and other formats you're likely already
seeing and using in your structured data.
Additionally, Miller gives you the option to include comments within your data.
@ -362,6 +362,16 @@ GENMD-RUN-COMMAND
mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt
GENMD-EOF
## DCF (Debian control file)
GENMD-RUN-COMMAND
cat data/sample.dcf
GENMD-EOF
GENMD-RUN-COMMAND
mlr -i dcf -o json cat data/sample.dcf
GENMD-EOF
## Data-conversion keystroke-savers
While you can do format conversion using `mlr --icsv --ojson cat myfile.csv`, there are also keystroke-savers for this purpose, such as `mlr --c2j cat myfile.csv`. For a complete list:

View file

@ -468,7 +468,7 @@ trying to define a [local variable](#local-variable) `if = 3` will result in a p
A subsequence of a text file in between line-ending symbols such as the special linefeed character.
Tools in the [Unix toolkit](#unix-toolkit) generally operate on lines; Miller is designed to do
that (using the [NIDX format flags](file-formats.md#nidx-index-numbered-toolkit-style)), as well
as non-line-oriented formats such as [CSV, TSV, JSON, YAML, and others](file-formats.md).
as non-line-oriented formats such as [CSV, TSV, JSON, YAML, DCF, and others](file-formats.md).
## local variable

View file

@ -452,7 +452,7 @@ trying to define a [local variable](#local-variable) `if = 3` will result in a p
A subsequence of a text file in between line-ending symbols such as the special linefeed character.
Tools in the [Unix toolkit](#unix-toolkit) generally operate on lines; Miller is designed to do
that (using the [NIDX format flags](file-formats.md#nidx-index-numbered-toolkit-style)), as well
as non-line-oriented formats such as [CSV, TSV, JSON, YAML, and others](file-formats.md).
as non-line-oriented formats such as [CSV, TSV, JSON, YAML, DCF, and others](file-formats.md).
## local variable

View file

@ -16,7 +16,7 @@ Quick links:
</div>
# Introduction
**Miller is a command-line tool for querying, shaping, and reformatting data files in various formats, including CSV, TSV, JSON, JSON Lines, and YAML.**
**Miller is a command-line tool for querying, shaping, and reformatting data files in various formats, including CSV, TSV, JSON, JSON Lines, YAML, and DCF.**
**The big picture:** Even well into the 21st century, our world is full of text-formatted data such as CSV. Google _CSV memes_, for example. We need tooling to _thrive in this world_, nimbly manipulating data which is in CSVs. And we need tooling to _move beyond CSV_, to be able to pull data out and into other storage and processing systems. Miller is designed for both of these goals.
@ -39,7 +39,7 @@ pretty-print your data horizontally or vertically to make it easier to read.
<b>mlr --json head -n 1 myfile.json</b>
</pre>
The `sort`, `head`, etc., are called *verbs*. They're analogs of familiar command-line tools like `sort`, `head`, and so on -- but they're aware of name-indexed, multi-line file formats like CSV, TSV, JSON, and YAML. In addition, though, using Miller's `put` verb, you can use programming-language statements for expressions like
The `sort`, `head`, etc., are called *verbs*. They're analogs of familiar command-line tools like `sort`, `head`, and so on -- but they're aware of name-indexed, multi-line file formats like CSV, TSV, JSON, YAML, and DCF. In addition, though, using Miller's `put` verb, you can use programming-language statements for expressions like
<pre class="pre-highlight-non-pair">
<b>mlr --csv put '$rate = $units / $seconds' input.csv</b>

View file

@ -1,6 +1,6 @@
# Introduction
**Miller is a command-line tool for querying, shaping, and reformatting data files in various formats, including CSV, TSV, JSON, JSON Lines, and YAML.**
**Miller is a command-line tool for querying, shaping, and reformatting data files in various formats, including CSV, TSV, JSON, JSON Lines, YAML, and DCF.**
**The big picture:** Even well into the 21st century, our world is full of text-formatted data such as CSV. Google _CSV memes_, for example. We need tooling to _thrive in this world_, nimbly manipulating data which is in CSVs. And we need tooling to _move beyond CSV_, to be able to pull data out and into other storage and processing systems. Miller is designed for both of these goals.
@ -23,7 +23,7 @@ GENMD-SHOW-COMMAND
mlr --json head -n 1 myfile.json
GENMD-EOF
The `sort`, `head`, etc., are called *verbs*. They're analogs of familiar command-line tools like `sort`, `head`, and so on -- but they're aware of name-indexed, multi-line file formats like CSV, TSV, JSON, and YAML. In addition, though, using Miller's `put` verb, you can use programming-language statements for expressions like
The `sort`, `head`, etc., are called *verbs*. They're analogs of familiar command-line tools like `sort`, `head`, and so on -- but they're aware of name-indexed, multi-line file formats like CSV, TSV, JSON, YAML, and DCF. In addition, though, using Miller's `put` verb, you can use programming-language statements for expressions like
GENMD-SHOW-COMMAND
mlr --csv put '$rate = $units / $seconds' input.csv

View file

@ -141,6 +141,17 @@ This is simply a copy of what you should see on running `man mlr` at a command p
| fox jumped | Record 2: "1":"fox", "2":"jumped"
+---------------------+
DCF: Debian control file format
+------------+
| apple: 1 |
| bat: 2 |
| cog: 3 |
| |
| dish: 7 |
| egg: 8 |
| 3: flint |
+------------+
1mHELP OPTIONS0m
Type 'mlr help {topic}' for any of the following:
Essentials:
@ -393,6 +404,8 @@ This is simply a copy of what you should see on running `man mlr` at a command p
--asv or --asvlite Use ASV format for input and output data.
--csv or -c or --c2c Use CSV format for input and output data.
--csvlite Use CSV-lite format for input and output data.
--dcf Use Debian control file (DCF) format for input and
output data.
--dkvp or --d2d Use DKVP format for input and output data.
--gen-field-name Specify field name for --igen. Defaults to "i".
--gen-start Specify start value for --igen. Defaults to 1.
@ -401,6 +414,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
--iasv or --iasvlite Use ASV format for input data.
--icsv Use CSV format for input data.
--icsvlite Use CSV-lite format for input data.
--idcf Use Debian control file (DCF) format for input data.
--idkvp Use DKVP format for input data.
--igen Ignore input files and instead generate sequential
numeric input using --gen-field-name, --gen-start,
@ -424,6 +438,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
--oasv or --oasvlite Use ASV format for output data.
--ocsv Use CSV format for output data.
--ocsvlite Use CSV-lite format for output data.
--odcf Use Debian control file (DCF) format for output data.
--odkvp Use DKVP format for output data.
--ojson Use JSON format for output data.
--ojsonl Use JSON Lines format for output data.
@ -470,7 +485,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
1mFORMAT-CONVERSION KEYSTROKE-SAVER FLAGS0m
As keystroke-savers for format-conversion you may use the following.
The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines,
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively.
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out).
| In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML |
+----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+
@ -837,6 +852,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
Format FS PS RS
csv "," N/A "\n"
csvlite "," N/A "\n"
dcf N/A N/A N/A
dkvp "," "=" "\n"
gen "," N/A "\n"
json N/A N/A N/A
@ -3787,5 +3803,5 @@ This is simply a copy of what you should see on running `man mlr` at a command p
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-02-15 4mMILLER24m(1)
2026-02-16 4mMILLER24m(1)
</pre>

View file

@ -120,6 +120,17 @@
| fox jumped | Record 2: "1":"fox", "2":"jumped"
+---------------------+
DCF: Debian control file format
+------------+
| apple: 1 |
| bat: 2 |
| cog: 3 |
| |
| dish: 7 |
| egg: 8 |
| 3: flint |
+------------+
1mHELP OPTIONS0m
Type 'mlr help {topic}' for any of the following:
Essentials:
@ -372,6 +383,8 @@
--asv or --asvlite Use ASV format for input and output data.
--csv or -c or --c2c Use CSV format for input and output data.
--csvlite Use CSV-lite format for input and output data.
--dcf Use Debian control file (DCF) format for input and
output data.
--dkvp or --d2d Use DKVP format for input and output data.
--gen-field-name Specify field name for --igen. Defaults to "i".
--gen-start Specify start value for --igen. Defaults to 1.
@ -380,6 +393,7 @@
--iasv or --iasvlite Use ASV format for input data.
--icsv Use CSV format for input data.
--icsvlite Use CSV-lite format for input data.
--idcf Use Debian control file (DCF) format for input data.
--idkvp Use DKVP format for input data.
--igen Ignore input files and instead generate sequential
numeric input using --gen-field-name, --gen-start,
@ -403,6 +417,7 @@
--oasv or --oasvlite Use ASV format for output data.
--ocsv Use CSV format for output data.
--ocsvlite Use CSV-lite format for output data.
--odcf Use Debian control file (DCF) format for output data.
--odkvp Use DKVP format for output data.
--ojson Use JSON format for output data.
--ojsonl Use JSON Lines format for output data.
@ -449,7 +464,7 @@
1mFORMAT-CONVERSION KEYSTROKE-SAVER FLAGS0m
As keystroke-savers for format-conversion you may use the following.
The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines,
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively.
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out).
| In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML |
+----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+
@ -816,6 +831,7 @@
Format FS PS RS
csv "," N/A "\n"
csvlite "," N/A "\n"
dcf N/A N/A N/A
dkvp "," "=" "\n"
gen "," N/A "\n"
json N/A N/A N/A
@ -3766,4 +3782,4 @@
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-02-15 4mMILLER24m(1)
2026-02-16 4mMILLER24m(1)

View file

@ -16,7 +16,7 @@ for section_name in section_names
# The markdown in this section looks a lot better when hand-crafted (thanks Nikos!).
puts <<EOF
The letters `c`, `t`, `j`, `l`, `d`, `n`, `x`, `p`, `m`, and `y` refer to formats CSV, TSV, JSON, JSON Lines,
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. Note that markdown format is available for
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use `--dcf` for DCF in and out). Note that markdown format is available for
output only.
| In \\ out | **CSV** | **TSV** | **JSON** | **JSONL** | **DKVP** | **NIDX** | **XTAB** | **PPRINT** | **Markdown** | **YAML** |

View file

@ -47,7 +47,7 @@ See also [Differences from other programming languages](reference-dsl-difference
Miller's input and output are all text-oriented: all the
[file formats supported by Miller](file-formats.md) are human-readable text,
such as CSV, TSV, and JSON; binary formats such as
such as CSV, TSV, JSON, and DCF; binary formats such as
[BSON](https://bsonspec.org/) and [Parquet](https://parquet.apache.org/) are
not supported (as of mid-2021). In this sense, everything is a string in and out of
Miller -- be it in data files, or in DSL expressions you key in.

View file

@ -31,7 +31,7 @@ See also [Differences from other programming languages](reference-dsl-difference
Miller's input and output are all text-oriented: all the
[file formats supported by Miller](file-formats.md) are human-readable text,
such as CSV, TSV, and JSON; binary formats such as
such as CSV, TSV, JSON, and DCF; binary formats such as
[BSON](https://bsonspec.org/) and [Parquet](https://parquet.apache.org/) are
not supported (as of mid-2021). In this sense, everything is a string in and out of
Miller -- be it in data files, or in DSL expressions you key in.

View file

@ -155,6 +155,7 @@ are overridden in all cases by setting output format to `format2`.
* `--asv or --asvlite`: Use ASV format for input and output data.
* `--csv or -c or --c2c`: Use CSV format for input and output data.
* `--csvlite`: Use CSV-lite format for input and output data.
* `--dcf`: Use Debian control file (DCF) format for input and output data.
* `--dkvp or --d2d`: Use DKVP format for input and output data.
* `--gen-field-name`: Specify field name for --igen. Defaults to "i".
* `--gen-start`: Specify start value for --igen. Defaults to 1.
@ -163,6 +164,7 @@ are overridden in all cases by setting output format to `format2`.
* `--iasv or --iasvlite`: Use ASV format for input data.
* `--icsv`: Use CSV format for input data.
* `--icsvlite`: Use CSV-lite format for input data.
* `--idcf`: Use Debian control file (DCF) format for input data.
* `--idkvp`: Use DKVP format for input data.
* `--igen`: Ignore input files and instead generate sequential numeric input using --gen-field-name, --gen-start, --gen-step, and --gen-stop values. See also the seqgen verb, which is more useful/intuitive.
* `--ijson`: Use JSON format for input data.
@ -182,6 +184,7 @@ are overridden in all cases by setting output format to `format2`.
* `--oasv or --oasvlite`: Use ASV format for output data.
* `--ocsv`: Use CSV format for output data.
* `--ocsvlite`: Use CSV-lite format for output data.
* `--odcf`: Use Debian control file (DCF) format for output data.
* `--odkvp`: Use DKVP format for output data.
* `--ojson`: Use JSON format for output data.
* `--ojsonl`: Use JSON Lines format for output data.
@ -219,7 +222,7 @@ See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatt
## Format-conversion keystroke-saver flags
The letters `c`, `t`, `j`, `l`, `d`, `n`, `x`, `p`, `m`, and `y` refer to formats CSV, TSV, JSON, JSON Lines,
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. Note that markdown format is available for
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use `--dcf` for DCF in and out). Note that markdown format is available for
output only.
| In \ out | **CSV** | **TSV** | **JSON** | **JSONL** | **DKVP** | **NIDX** | **XTAB** | **PPRINT** | **Markdown** | **YAML** |
@ -494,6 +497,7 @@ Notes about all other separators:
Format FS PS RS
csv "," N/A "\n"
csvlite "," N/A "\n"
dcf N/A N/A N/A
dkvp "," "=" "\n"
gen "," N/A "\n"
json N/A N/A N/A

View file

@ -276,6 +276,7 @@ Notes:
| [**TSV-lite**](file-formats.md#csvtsvasvusvetc) | Default `\n` * | Default `\t` | None |
| [**JSON**](file-formats.md#json) | N/A; records are between `{` and `}` | Always `,`; not alterable | Always `:`; not alterable |
| [**YAML**](file-formats.md#yaml) | N/A; documents separated by `---` or single array | N/A; not alterable | Always `:`; not alterable |
| [**DCF**](file-formats.md#dcf-debian-control-file) | N/A; paragraphs separated by blank lines | N/A; not alterable | Always `:`; not alterable |
| [**DKVP**](file-formats.md#dkvp-key-value-pairs) | Default `\n` | Default `,` | Default `=` |
| [**NIDX**](file-formats.md#nidx-index-numbered-toolkit-style) | Default `\n` | Default space | None |
| [**XTAB**](file-formats.md#xtab-vertical-tabular) | Not used; records are separated by an extra FS | `\n` * | Default: space with repeats |

View file

@ -166,6 +166,7 @@ Notes:
| [**TSV-lite**](file-formats.md#csvtsvasvusvetc) | Default `\n` * | Default `\t` | None |
| [**JSON**](file-formats.md#json) | N/A; records are between `{` and `}` | Always `,`; not alterable | Always `:`; not alterable |
| [**YAML**](file-formats.md#yaml) | N/A; documents separated by `---` or single array | N/A; not alterable | Always `:`; not alterable |
| [**DCF**](file-formats.md#dcf-debian-control-file) | N/A; paragraphs separated by blank lines | N/A; not alterable | Always `:`; not alterable |
| [**DKVP**](file-formats.md#dkvp-key-value-pairs) | Default `\n` | Default `,` | Default `=` |
| [**NIDX**](file-formats.md#nidx-index-numbered-toolkit-style) | Default `\n` | Default space | None |
| [**XTAB**](file-formats.md#xtab-vertical-tabular) | Not used; records are separated by an extra FS | `\n` * | Default: space with repeats |

3
go.mod
View file

@ -40,6 +40,9 @@ require (
github.com/google/pprof v0.0.0-20211214055906-6f57359322fd // indirect
github.com/kshedden/dstream v0.0.0-20190512025041-c4c410631beb // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
golang.org/x/crypto v0.40.0 // indirect
golang.org/x/tools v0.41.0 // indirect
gonum.org/v1/gonum v0.16.0 // indirect
pault.ag/go/debian v0.19.0 // indirect
pault.ag/go/topsort v0.1.1 // indirect
)

6
go.sum
View file

@ -41,6 +41,8 @@ github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
golang.org/x/crypto v0.40.0 h1:r4x+VvoG5Fm+eJcxMaY8CQM7Lb0l1lsmjGBQ6s8BfKM=
golang.org/x/crypto v0.40.0/go.mod h1:Qr1vMER5WyS2dfPHAlsOj01wgLbsyWtFn/aY+5+ZdxY=
golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
@ -58,3 +60,7 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
pault.ag/go/debian v0.19.0 h1:RUxCjScMbnlqFH5I+qsmyjZH8fXXtQ05rlkMJop3tjo=
pault.ag/go/debian v0.19.0/go.mod h1:1LMojDAazlJ7cA5Ne6H2ZHD4hh3o8NRiW+MpvQRji2o=
pault.ag/go/topsort v0.1.1 h1:L0QnhUly6LmTv0e3DEzbN2q6/FGgAcQvaEw65S53Bg4=
pault.ag/go/topsort v0.1.1/go.mod h1:r1kc/L0/FZ3HhjezBIPaNVhkqv8L0UJ9bxRuHRVZ0q4=

View file

@ -120,6 +120,17 @@
| fox jumped | Record 2: "1":"fox", "2":"jumped"
+---------------------+
DCF: Debian control file format
+------------+
| apple: 1 |
| bat: 2 |
| cog: 3 |
| |
| dish: 7 |
| egg: 8 |
| 3: flint |
+------------+
1mHELP OPTIONS0m
Type 'mlr help {topic}' for any of the following:
Essentials:
@ -372,6 +383,8 @@
--asv or --asvlite Use ASV format for input and output data.
--csv or -c or --c2c Use CSV format for input and output data.
--csvlite Use CSV-lite format for input and output data.
--dcf Use Debian control file (DCF) format for input and
output data.
--dkvp or --d2d Use DKVP format for input and output data.
--gen-field-name Specify field name for --igen. Defaults to "i".
--gen-start Specify start value for --igen. Defaults to 1.
@ -380,6 +393,7 @@
--iasv or --iasvlite Use ASV format for input data.
--icsv Use CSV format for input data.
--icsvlite Use CSV-lite format for input data.
--idcf Use Debian control file (DCF) format for input data.
--idkvp Use DKVP format for input data.
--igen Ignore input files and instead generate sequential
numeric input using --gen-field-name, --gen-start,
@ -403,6 +417,7 @@
--oasv or --oasvlite Use ASV format for output data.
--ocsv Use CSV format for output data.
--ocsvlite Use CSV-lite format for output data.
--odcf Use Debian control file (DCF) format for output data.
--odkvp Use DKVP format for output data.
--ojson Use JSON format for output data.
--ojsonl Use JSON Lines format for output data.
@ -449,7 +464,7 @@
1mFORMAT-CONVERSION KEYSTROKE-SAVER FLAGS0m
As keystroke-savers for format-conversion you may use the following.
The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines,
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively.
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out).
| In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML |
+----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+
@ -816,6 +831,7 @@
Format FS PS RS
csv "," N/A "\n"
csvlite "," N/A "\n"
dcf N/A N/A N/A
dkvp "," "=" "\n"
gen "," N/A "\n"
json N/A N/A N/A
@ -3766,4 +3782,4 @@
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
https://miller.readthedocs.io
2026-02-15 4mMILLER24m(1)
2026-02-16 4mMILLER24m(1)

View file

@ -2,12 +2,12 @@
.\" Title: mlr
.\" Author: [see the "AUTHOR" section]
.\" Generator: ./mkman.rb
.\" Date: 2026-02-15
.\" Date: 2026-02-16
.\" Manual: \ \&
.\" Source: \ \&
.\" Language: English
.\"
.TH "MILLER" "1" "2026-02-15" "\ \&" "\ \&"
.TH "MILLER" "1" "2026-02-16" "\ \&" "\ \&"
.\" -----------------------------------------------------------------
.\" * Portability definitions
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -150,6 +150,17 @@ NIDX: implicitly numerically indexed (Unix-toolkit style)
| the quick brown | Record 1: "1":"the", "2":"quick", "3":"brown"
| fox jumped | Record 2: "1":"fox", "2":"jumped"
+---------------------+
DCF: Debian control file format
+------------+
| apple: 1 |
| bat: 2 |
| cog: 3 |
| |
| dish: 7 |
| egg: 8 |
| 3: flint |
+------------+
.fi
.if n \{\
.RE
@ -461,6 +472,8 @@ are overridden in all cases by setting output format to `format2`.
--asv or --asvlite Use ASV format for input and output data.
--csv or -c or --c2c Use CSV format for input and output data.
--csvlite Use CSV-lite format for input and output data.
--dcf Use Debian control file (DCF) format for input and
output data.
--dkvp or --d2d Use DKVP format for input and output data.
--gen-field-name Specify field name for --igen. Defaults to "i".
--gen-start Specify start value for --igen. Defaults to 1.
@ -469,6 +482,7 @@ are overridden in all cases by setting output format to `format2`.
--iasv or --iasvlite Use ASV format for input data.
--icsv Use CSV format for input data.
--icsvlite Use CSV-lite format for input data.
--idcf Use Debian control file (DCF) format for input data.
--idkvp Use DKVP format for input data.
--igen Ignore input files and instead generate sequential
numeric input using --gen-field-name, --gen-start,
@ -492,6 +506,7 @@ are overridden in all cases by setting output format to `format2`.
--oasv or --oasvlite Use ASV format for output data.
--ocsv Use CSV format for output data.
--ocsvlite Use CSV-lite format for output data.
--odcf Use Debian control file (DCF) format for output data.
--odkvp Use DKVP format for output data.
--ojson Use JSON format for output data.
--ojsonl Use JSON Lines format for output data.
@ -554,7 +569,7 @@ See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatt
.nf
As keystroke-savers for format-conversion you may use the following.
The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines,
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively.
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out).
| In\eout | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML |
+----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+
@ -977,6 +992,7 @@ Notes about all other separators:
Format FS PS RS
csv "," N/A "\en"
csvlite "," N/A "\en"
dcf N/A N/A N/A
dkvp "," "=" "\en"
gen "," N/A "\en"
json N/A N/A N/A

View file

@ -61,7 +61,8 @@ package cli
func DecideFinalFlatten(writerOptions *TWriterOptions) bool {
ofmt := writerOptions.OutputFileFormat
if writerOptions.AutoFlatten {
if ofmt != "json" {
// Preserve nested/array structure for formats that support it.
if ofmt != "json" && ofmt != "jsonl" && ofmt != "dcf" {
return true
}
}

View file

@ -787,6 +787,15 @@ var FileFormatFlagSection = FlagSection{
},
},
{
name: "--idcf",
help: "Use Debian control file (DCF) format for input data.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
options.ReaderOptions.InputFileFormat = "dcf"
*pargi += 1
},
},
{
name: "--inidx",
help: "Use NIDX format for input data.",
@ -1021,6 +1030,15 @@ var FileFormatFlagSection = FlagSection{
},
},
{
name: "--odcf",
help: "Use Debian control file (DCF) format for output data.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
options.WriterOptions.OutputFileFormat = "dcf"
*pargi += 1
},
},
{
name: "--onidx",
help: "Use NIDX format for output data.",
@ -1200,6 +1218,16 @@ var FileFormatFlagSection = FlagSection{
},
},
{
name: "--dcf",
help: "Use Debian control file (DCF) format for input and output data.",
parser: func(args []string, argc int, pargi *int, options *TOptions) {
options.ReaderOptions.InputFileFormat = "dcf"
options.WriterOptions.OutputFileFormat = "dcf"
*pargi += 1
},
},
{
name: "--nidx",
help: "Use NIDX format for input and output data.",
@ -1252,7 +1280,7 @@ var FileFormatFlagSection = FlagSection{
func FormatConversionKeystrokeSaverPrintInfo() {
fmt.Println(`As keystroke-savers for format-conversion you may use the following.
The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines,
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively.
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out).
| In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML |
+----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+

View file

@ -88,6 +88,7 @@ var defaultFSes = map[string]string{
"dkvp": ",",
"json": "N/A", // not alterable; not parameterizable in JSON format
"yaml": "N/A",
"dcf": "N/A",
"nidx": " ",
"markdown": " ",
"pprint": " ",
@ -102,6 +103,7 @@ var defaultPSes = map[string]string{
"dkvp": "=",
"json": "N/A", // not alterable; not parameterizable in JSON format
"yaml": "N/A",
"dcf": "N/A",
"markdown": "N/A",
"nidx": "N/A",
"pprint": "N/A",
@ -116,6 +118,7 @@ var defaultRSes = map[string]string{
"dkvp": "\n",
"json": "N/A", // not alterable; not parameterizable in JSON format
"yaml": "N/A",
"dcf": "N/A",
"markdown": "\n",
"nidx": "\n",
"pprint": "\n",
@ -130,6 +133,7 @@ var defaultAllowRepeatIFSes = map[string]bool{
"dkvp": false,
"json": false,
"yaml": false,
"dcf": false,
"markdown": false,
"nidx": false,
"pprint": true,

View file

@ -0,0 +1,198 @@
package input
import (
"io"
"strings"
"pault.ag/go/debian/control"
"github.com/johnkerl/miller/v6/pkg/cli"
"github.com/johnkerl/miller/v6/pkg/lib"
"github.com/johnkerl/miller/v6/pkg/mlrval"
"github.com/johnkerl/miller/v6/pkg/types"
)
// DCF list-valued field names (comma-separated in the format). These are
// exposed as Miller arrays; all other fields remain strings.
var dcfListFieldNames = map[string]bool{
"Depends": true,
"Pre-Depends": true,
"Recommends": true,
"Suggests": true,
"Enhances": true,
"Breaks": true,
"Conflicts": true,
"Replaces": true,
"Build-Depends": true,
"Build-Depends-Indep": true,
"Build-Conflicts": true,
"Build-Conflicts-Indep": true,
"Built-Using": true,
}
type RecordReaderDCF struct {
readerOptions *cli.TReaderOptions
recordsPerBatch int64
}
func NewRecordReaderDCF(
readerOptions *cli.TReaderOptions,
recordsPerBatch int64,
) (*RecordReaderDCF, error) {
return &RecordReaderDCF{
readerOptions: readerOptions,
recordsPerBatch: recordsPerBatch,
}, nil
}
func (reader *RecordReaderDCF) Read(
filenames []string,
context types.Context,
readerChannel chan<- []*types.RecordAndContext,
errorChannel chan error,
downstreamDoneChannel <-chan bool,
) {
if filenames != nil {
if len(filenames) == 0 {
handle, err := lib.OpenStdin(
reader.readerOptions.Prepipe,
reader.readerOptions.PrepipeIsRaw,
reader.readerOptions.FileInputEncoding,
)
if err != nil {
errorChannel <- err
} else {
reader.processHandle(handle, "(stdin)", &context, readerChannel, errorChannel, downstreamDoneChannel)
}
} else {
for _, filename := range filenames {
handle, err := lib.OpenFileForRead(
filename,
reader.readerOptions.Prepipe,
reader.readerOptions.PrepipeIsRaw,
reader.readerOptions.FileInputEncoding,
)
if err != nil {
errorChannel <- err
} else {
reader.processHandle(handle, filename, &context, readerChannel, errorChannel, downstreamDoneChannel)
handle.Close()
}
}
}
}
readerChannel <- types.NewEndOfStreamMarkerList(&context)
}
func (reader *RecordReaderDCF) processHandle(
handle io.Reader,
filename string,
context *types.Context,
readerChannel chan<- []*types.RecordAndContext,
errorChannel chan error,
downstreamDoneChannel <-chan bool,
) {
context.UpdateForStartOfFile(filename)
recordsPerBatch := reader.recordsPerBatch
pr, err := control.NewParagraphReader(handle, nil)
if err != nil {
errorChannel <- err
return
}
recordsAndContexts := make([]*types.RecordAndContext, 0, recordsPerBatch)
i := int64(0)
for {
i++
if i%recordsPerBatch == 0 {
select {
case <-downstreamDoneChannel:
goto flush
default:
}
}
para, err := pr.Next()
if err != nil {
if err == io.EOF {
break
}
errorChannel <- err
return
}
if para == nil {
break
}
rec := dcfParagraphToRecord(para, reader.readerOptions.DedupeFieldNames)
if rec == nil {
// dedupe or other error already sent
return
}
context.UpdateForInputRecord()
recordsAndContexts = append(recordsAndContexts, types.NewRecordAndContext(rec, context))
if int64(len(recordsAndContexts)) >= recordsPerBatch {
readerChannel <- recordsAndContexts
recordsAndContexts = make([]*types.RecordAndContext, 0, recordsPerBatch)
}
}
flush:
if len(recordsAndContexts) > 0 {
readerChannel <- recordsAndContexts
}
}
// splitCommaList splits a DCF comma-separated value and returns Miller Mlrval
// array elements (trimmed, skipping empty).
func splitCommaList(s string) []*mlrval.Mlrval {
parts := strings.Split(s, ",")
out := make([]*mlrval.Mlrval, 0, len(parts))
for _, p := range parts {
p = strings.TrimSpace(p)
if p == "" {
continue
}
out = append(out, mlrval.FromString(p))
}
return out
}
// dcfParagraphToRecord converts a control paragraph to a Miller record. List
// fields (Depends, Build-Depends, etc.) become arrays; all other fields stay
// strings. Uses para.Order to preserve field order. Returns nil on error (caller
// must have sent on errorChannel).
func dcfParagraphToRecord(para *control.Paragraph, dedupeFieldNames bool) *mlrval.Mlrmap {
rec := mlrval.NewMlrmapAsRecord()
// Preserve order from the DCF file when present.
keys := para.Order
if len(keys) == 0 {
for k := range para.Values {
keys = append(keys, k)
}
}
for _, k := range keys {
v, ok := para.Values[k]
if !ok {
continue
}
var mv *mlrval.Mlrval
if dcfListFieldNames[k] {
mv = mlrval.FromArray(splitCommaList(v))
} else {
mv = mlrval.FromString(v)
}
if _, err := rec.PutReferenceMaybeDedupe(k, mv, dedupeFieldNames); err != nil {
return nil
}
}
return rec
}

View file

@ -30,6 +30,8 @@ func Create(readerOptions *cli.TReaderOptions, recordsPerBatch int64) (IRecordRe
return NewRecordReaderTSV(readerOptions, recordsPerBatch)
case "xtab":
return NewRecordReaderXTAB(readerOptions, recordsPerBatch)
case "dcf":
return NewRecordReaderDCF(readerOptions, recordsPerBatch)
case "gen":
return NewPseudoReaderGen(readerOptions, recordsPerBatch)
default:

View file

@ -2,7 +2,7 @@
// Wrapper for os.Open which maps string filename to *os.File, which in turn
// implements io.ReadCloser, and optional in turn wrapping that in a
// gzip/zlib/bunzip2 reader. Shared across record-readers for all the various
// input-file formats (CSV, JSON, XTAB, DKVP, NIDX, PPRINT) which Miller
// input-file formats (CSV, JSON, XTAB, DKVP, NIDX, PPRINT, DCF) which Miller
// supports.
//
// There are two ways of handling compressed data in the Miller Go port:

View file

@ -0,0 +1,85 @@
package output
import (
"bufio"
"strings"
"github.com/johnkerl/miller/v6/pkg/cli"
"github.com/johnkerl/miller/v6/pkg/colorizer"
"github.com/johnkerl/miller/v6/pkg/mlrval"
"github.com/johnkerl/miller/v6/pkg/types"
)
type RecordWriterDCF struct {
writerOptions *cli.TWriterOptions
}
func NewRecordWriterDCF(writerOptions *cli.TWriterOptions) (*RecordWriterDCF, error) {
return &RecordWriterDCF{
writerOptions: writerOptions,
}, nil
}
func (writer *RecordWriterDCF) Write(
outrec *mlrval.Mlrmap,
_ *types.Context,
bufferedOutputStream *bufio.Writer,
outputIsStdout bool,
) error {
if outrec == nil {
return nil
}
if outrec.IsEmpty() {
bufferedOutputStream.WriteString("\n")
return nil
}
for pe := outrec.Head; pe != nil; pe = pe.Next {
valueStr := dcfValueString(pe.Value)
keyStr := colorizer.MaybeColorizeKey(pe.Key, outputIsStdout)
valueStr = colorizer.MaybeColorizeValue(valueStr, outputIsStdout)
writeDCFField(bufferedOutputStream, keyStr, valueStr)
}
bufferedOutputStream.WriteString("\n")
return nil
}
// dcfValueString returns the string form of a field value for DCF output.
// Arrays are joined with ", " to match list fields (Depends, etc.).
func dcfValueString(mv *mlrval.Mlrval) string {
if mv == nil {
return ""
}
if mv.IsArray() {
arr := mv.GetArray()
if arr == nil || len(arr) == 0 {
return ""
}
parts := make([]string, 0, len(arr))
for _, el := range arr {
if el != nil {
parts = append(parts, el.String())
}
}
return strings.Join(parts, ", ")
}
return mv.String()
}
// writeDCFField writes one "Key: value" line, folding on newlines per DCF:
// continuation lines start with a single space.
func writeDCFField(b *bufio.Writer, key, value string) {
lines := strings.Split(value, "\n")
for i, line := range lines {
if i == 0 {
b.WriteString(key)
b.WriteString(": ")
b.WriteString(line)
b.WriteString("\n")
} else {
b.WriteString(" ")
b.WriteString(line)
b.WriteString("\n")
}
}
}

View file

@ -20,6 +20,8 @@ func Create(writerOptions *cli.TWriterOptions) (IRecordWriter, error) {
return NewRecordWriterJSONLines(writerOptions)
case "yaml":
return NewRecordWriterYAML(writerOptions)
case "dcf":
return NewRecordWriterDCF(writerOptions)
case "md":
return NewRecordWriterMarkdown(writerOptions)
case "markdown":

View file

@ -447,6 +447,17 @@ NIDX: implicitly numerically indexed (Unix-toolkit style)
| the quick brown | Record 1: "1":"the", "2":"quick", "3":"brown"
| fox jumped | Record 2: "1":"fox", "2":"jumped"
+---------------------+
DCF: Debian control file format
+------------+
| apple: 1 |
| bat: 2 |
| cog: 3 |
| |
| dish: 7 |
| egg: 8 |
| 3: flint |
+------------+
`)
}

View file

@ -0,0 +1 @@
mlr --itsv --rs lf --odcf cat test/input/simple.tsv

View file

View file

@ -0,0 +1,30 @@
a: pan
b: pan
i: 1
x: 2
y: 0.98994500
a: eks
b: pan
i: 2
x: 1
y: 0.77515900
a: wye
b: wye
i: 3
x: 1
y: 0.76164200
a: eks
b: wye
i: 4
x: 5
y: 0.32293400
a: wye
b: pan
i: 5
x: 5
y: 0.44828300

View file

@ -0,0 +1 @@
mlr --icsv --odcf cat ${CASEDIR}/input

View file

View file

@ -0,0 +1,4 @@
a: 1
b: 2|3
c: 4

View file

@ -0,0 +1,2 @@
a,b,c
1,2|3,4

View file

@ -0,0 +1 @@
mlr --idcf --odcf cat test/input/test.dcf

View file

View file

@ -0,0 +1,10 @@
Package: foo
Version: 1.0
Depends: libc6 (>= 2.0), libfoo (>= 1.2)
Description: A test package.
Package: bar
Version: 2.0
Recommends: foo
Description: Another package.

View file

@ -0,0 +1 @@
mlr --idcf --ojson cat test/input/test.dcf

View file

View file

@ -0,0 +1,14 @@
[
{
"Package": "foo",
"Version": "1.0",
"Depends": ["libc6 (>= 2.0)", "libfoo (>= 1.2)"],
"Description": "A test package."
},
{
"Package": "bar",
"Version": "2.0",
"Recommends": ["foo"],
"Description": "Another package."
}
]

9
test/input/test.dcf Normal file
View file

@ -0,0 +1,9 @@
Package: foo
Version: 1.0
Depends: libc6 (>= 2.0), libfoo (>= 1.2)
Description: A test package.
Package: bar
Version: 2.0
Recommends: foo
Description: Another package.