mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-17 16:38:54 +00:00
Add DCF (Debian Control Format) as supported file type (#1970)
* reader * record writer * pkg/output/record_writer_dcf.go * test/input/test.dcf * test/cases/io-dcf/ * pkg/cli/option_parse.go * make fmt * make dev * docs and `make dev`
This commit is contained in:
parent
a5f4c9d895
commit
daae7ff7f0
44 changed files with 579 additions and 29 deletions
9
docs/src/data/sample.dcf
Normal file
9
docs/src/data/sample.dcf
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
Package: foo
|
||||
Version: 1.0
|
||||
Depends: libc6 (>= 2.0), libfoo (>= 1.2)
|
||||
Description: A test package.
|
||||
|
||||
Package: bar
|
||||
Version: 2.0
|
||||
Recommends: foo
|
||||
Description: Another package.
|
||||
|
|
@ -17,7 +17,7 @@ Quick links:
|
|||
# Features
|
||||
|
||||
Miller is like awk, sed, cut, join, and sort for **name-indexed data, such as
|
||||
CSV, TSV, JSON, JSON Lines, and YAML**. You get to work with your data using named
|
||||
CSV, TSV, JSON, JSON Lines, YAML, and DCF**. You get to work with your data using named
|
||||
fields, without needing to count positional column indices.
|
||||
|
||||
This is something the Unix toolkit always could have done, and arguably always
|
||||
|
|
@ -25,7 +25,7 @@ should have done. It operates on key-value-pair data while the familiar Unix
|
|||
tools operate on integer-indexed fields: if the natural data structure for the
|
||||
latter is the array, then Miller's natural data structure is the
|
||||
insertion-ordered hash map. This encompasses a **variety of data formats**,
|
||||
including but not limited to the familiar CSV, TSV, JSON, JSON Lines, and YAML.
|
||||
including but not limited to the familiar CSV, TSV, JSON, JSON Lines, YAML, and DCF.
|
||||
(Miller can handle **positionally-indexed data** as a special case.)
|
||||
|
||||
* Miller is **multi-purpose**: it's useful for **data cleaning**, **data reduction**, **statistical reporting**, **devops**, **system administration**, **log-file processing**, **format conversion**, and **database-query post-processing**.
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# Features
|
||||
|
||||
Miller is like awk, sed, cut, join, and sort for **name-indexed data, such as
|
||||
CSV, TSV, JSON, JSON Lines, and YAML**. You get to work with your data using named
|
||||
CSV, TSV, JSON, JSON Lines, YAML, and DCF**. You get to work with your data using named
|
||||
fields, without needing to count positional column indices.
|
||||
|
||||
This is something the Unix toolkit always could have done, and arguably always
|
||||
|
|
@ -9,7 +9,7 @@ should have done. It operates on key-value-pair data while the familiar Unix
|
|||
tools operate on integer-indexed fields: if the natural data structure for the
|
||||
latter is the array, then Miller's natural data structure is the
|
||||
insertion-ordered hash map. This encompasses a **variety of data formats**,
|
||||
including but not limited to the familiar CSV, TSV, JSON, JSON Lines, and YAML.
|
||||
including but not limited to the familiar CSV, TSV, JSON, JSON Lines, YAML, and DCF.
|
||||
(Miller can handle **positionally-indexed data** as a special case.)
|
||||
|
||||
* Miller is **multi-purpose**: it's useful for **data cleaning**, **data reduction**, **statistical reporting**, **devops**, **system administration**, **log-file processing**, **format conversion**, and **database-query post-processing**.
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ Quick links:
|
|||
# File formats
|
||||
|
||||
Miller handles name-indexed data using several formats: some you probably know
|
||||
by name, such as CSV, TSV, JSON, JSON Lines, and YAML -- and other formats you're likely already
|
||||
by name, such as CSV, TSV, JSON, JSON Lines, YAML, and DCF -- and other formats you're likely already
|
||||
seeing and using in your structured data.
|
||||
|
||||
Additionally, Miller gives you the option to include comments within your data.
|
||||
|
|
@ -108,6 +108,17 @@ NIDX: implicitly numerically indexed (Unix-toolkit style)
|
|||
| the quick brown | Record 1: "1":"the", "2":"quick", "3":"brown"
|
||||
| fox jumped | Record 2: "1":"fox", "2":"jumped"
|
||||
+---------------------+
|
||||
|
||||
DCF: Debian control file format
|
||||
+------------+
|
||||
| apple: 1 |
|
||||
| bat: 2 |
|
||||
| cog: 3 |
|
||||
| |
|
||||
| dish: 7 |
|
||||
| egg: 8 |
|
||||
| 3: flint |
|
||||
+------------+
|
||||
</pre>
|
||||
|
||||
## CSV/TSV/ASV/USV/etc.
|
||||
|
|
@ -720,6 +731,43 @@ the dawn's
|
|||
light
|
||||
</pre>
|
||||
|
||||
## DCF (Debian control file)
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>cat data/sample.dcf</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
Package: foo
|
||||
Version: 1.0
|
||||
Depends: libc6 (>= 2.0), libfoo (>= 1.2)
|
||||
Description: A test package.
|
||||
|
||||
Package: bar
|
||||
Version: 2.0
|
||||
Recommends: foo
|
||||
Description: Another package.
|
||||
</pre>
|
||||
|
||||
<pre class="pre-highlight-in-pair">
|
||||
<b>mlr -i dcf -o json cat data/sample.dcf</b>
|
||||
</pre>
|
||||
<pre class="pre-non-highlight-in-pair">
|
||||
[
|
||||
{
|
||||
"Package": "foo",
|
||||
"Version": "1.0",
|
||||
"Depends": ["libc6 (>= 2.0)", "libfoo (>= 1.2)"],
|
||||
"Description": "A test package."
|
||||
},
|
||||
{
|
||||
"Package": "bar",
|
||||
"Version": "2.0",
|
||||
"Recommends": ["foo"],
|
||||
"Description": "Another package."
|
||||
}
|
||||
]
|
||||
</pre>
|
||||
|
||||
## Data-conversion keystroke-savers
|
||||
|
||||
While you can do format conversion using `mlr --icsv --ojson cat myfile.csv`, there are also keystroke-savers for this purpose, such as `mlr --c2j cat myfile.csv`. For a complete list:
|
||||
|
|
@ -731,7 +779,7 @@ While you can do format conversion using `mlr --icsv --ojson cat myfile.csv`, th
|
|||
FORMAT-CONVERSION KEYSTROKE-SAVER FLAGS
|
||||
As keystroke-savers for format-conversion you may use the following.
|
||||
The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines,
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively.
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out).
|
||||
|
||||
| In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML |
|
||||
+----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# File formats
|
||||
|
||||
Miller handles name-indexed data using several formats: some you probably know
|
||||
by name, such as CSV, TSV, JSON, JSON Lines, and YAML -- and other formats you're likely already
|
||||
by name, such as CSV, TSV, JSON, JSON Lines, YAML, and DCF -- and other formats you're likely already
|
||||
seeing and using in your structured data.
|
||||
|
||||
Additionally, Miller gives you the option to include comments within your data.
|
||||
|
|
@ -362,6 +362,16 @@ GENMD-RUN-COMMAND
|
|||
mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt
|
||||
GENMD-EOF
|
||||
|
||||
## DCF (Debian control file)
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
cat data/sample.dcf
|
||||
GENMD-EOF
|
||||
|
||||
GENMD-RUN-COMMAND
|
||||
mlr -i dcf -o json cat data/sample.dcf
|
||||
GENMD-EOF
|
||||
|
||||
## Data-conversion keystroke-savers
|
||||
|
||||
While you can do format conversion using `mlr --icsv --ojson cat myfile.csv`, there are also keystroke-savers for this purpose, such as `mlr --c2j cat myfile.csv`. For a complete list:
|
||||
|
|
|
|||
|
|
@ -468,7 +468,7 @@ trying to define a [local variable](#local-variable) `if = 3` will result in a p
|
|||
A subsequence of a text file in between line-ending symbols such as the special linefeed character.
|
||||
Tools in the [Unix toolkit](#unix-toolkit) generally operate on lines; Miller is designed to do
|
||||
that (using the [NIDX format flags](file-formats.md#nidx-index-numbered-toolkit-style)), as well
|
||||
as non-line-oriented formats such as [CSV, TSV, JSON, YAML, and others](file-formats.md).
|
||||
as non-line-oriented formats such as [CSV, TSV, JSON, YAML, DCF, and others](file-formats.md).
|
||||
|
||||
## local variable
|
||||
|
||||
|
|
|
|||
|
|
@ -452,7 +452,7 @@ trying to define a [local variable](#local-variable) `if = 3` will result in a p
|
|||
A subsequence of a text file in between line-ending symbols such as the special linefeed character.
|
||||
Tools in the [Unix toolkit](#unix-toolkit) generally operate on lines; Miller is designed to do
|
||||
that (using the [NIDX format flags](file-formats.md#nidx-index-numbered-toolkit-style)), as well
|
||||
as non-line-oriented formats such as [CSV, TSV, JSON, YAML, and others](file-formats.md).
|
||||
as non-line-oriented formats such as [CSV, TSV, JSON, YAML, DCF, and others](file-formats.md).
|
||||
|
||||
## local variable
|
||||
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ Quick links:
|
|||
</div>
|
||||
# Introduction
|
||||
|
||||
**Miller is a command-line tool for querying, shaping, and reformatting data files in various formats, including CSV, TSV, JSON, JSON Lines, and YAML.**
|
||||
**Miller is a command-line tool for querying, shaping, and reformatting data files in various formats, including CSV, TSV, JSON, JSON Lines, YAML, and DCF.**
|
||||
|
||||
**The big picture:** Even well into the 21st century, our world is full of text-formatted data such as CSV. Google _CSV memes_, for example. We need tooling to _thrive in this world_, nimbly manipulating data which is in CSVs. And we need tooling to _move beyond CSV_, to be able to pull data out and into other storage and processing systems. Miller is designed for both of these goals.
|
||||
|
||||
|
|
@ -39,7 +39,7 @@ pretty-print your data horizontally or vertically to make it easier to read.
|
|||
<b>mlr --json head -n 1 myfile.json</b>
|
||||
</pre>
|
||||
|
||||
The `sort`, `head`, etc., are called *verbs*. They're analogs of familiar command-line tools like `sort`, `head`, and so on -- but they're aware of name-indexed, multi-line file formats like CSV, TSV, JSON, and YAML. In addition, though, using Miller's `put` verb, you can use programming-language statements for expressions like
|
||||
The `sort`, `head`, etc., are called *verbs*. They're analogs of familiar command-line tools like `sort`, `head`, and so on -- but they're aware of name-indexed, multi-line file formats like CSV, TSV, JSON, YAML, and DCF. In addition, though, using Miller's `put` verb, you can use programming-language statements for expressions like
|
||||
|
||||
<pre class="pre-highlight-non-pair">
|
||||
<b>mlr --csv put '$rate = $units / $seconds' input.csv</b>
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Introduction
|
||||
|
||||
**Miller is a command-line tool for querying, shaping, and reformatting data files in various formats, including CSV, TSV, JSON, JSON Lines, and YAML.**
|
||||
**Miller is a command-line tool for querying, shaping, and reformatting data files in various formats, including CSV, TSV, JSON, JSON Lines, YAML, and DCF.**
|
||||
|
||||
**The big picture:** Even well into the 21st century, our world is full of text-formatted data such as CSV. Google _CSV memes_, for example. We need tooling to _thrive in this world_, nimbly manipulating data which is in CSVs. And we need tooling to _move beyond CSV_, to be able to pull data out and into other storage and processing systems. Miller is designed for both of these goals.
|
||||
|
||||
|
|
@ -23,7 +23,7 @@ GENMD-SHOW-COMMAND
|
|||
mlr --json head -n 1 myfile.json
|
||||
GENMD-EOF
|
||||
|
||||
The `sort`, `head`, etc., are called *verbs*. They're analogs of familiar command-line tools like `sort`, `head`, and so on -- but they're aware of name-indexed, multi-line file formats like CSV, TSV, JSON, and YAML. In addition, though, using Miller's `put` verb, you can use programming-language statements for expressions like
|
||||
The `sort`, `head`, etc., are called *verbs*. They're analogs of familiar command-line tools like `sort`, `head`, and so on -- but they're aware of name-indexed, multi-line file formats like CSV, TSV, JSON, YAML, and DCF. In addition, though, using Miller's `put` verb, you can use programming-language statements for expressions like
|
||||
|
||||
GENMD-SHOW-COMMAND
|
||||
mlr --csv put '$rate = $units / $seconds' input.csv
|
||||
|
|
|
|||
|
|
@ -141,6 +141,17 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
| fox jumped | Record 2: "1":"fox", "2":"jumped"
|
||||
+---------------------+
|
||||
|
||||
DCF: Debian control file format
|
||||
+------------+
|
||||
| apple: 1 |
|
||||
| bat: 2 |
|
||||
| cog: 3 |
|
||||
| |
|
||||
| dish: 7 |
|
||||
| egg: 8 |
|
||||
| 3: flint |
|
||||
+------------+
|
||||
|
||||
1mHELP OPTIONS0m
|
||||
Type 'mlr help {topic}' for any of the following:
|
||||
Essentials:
|
||||
|
|
@ -393,6 +404,8 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
--asv or --asvlite Use ASV format for input and output data.
|
||||
--csv or -c or --c2c Use CSV format for input and output data.
|
||||
--csvlite Use CSV-lite format for input and output data.
|
||||
--dcf Use Debian control file (DCF) format for input and
|
||||
output data.
|
||||
--dkvp or --d2d Use DKVP format for input and output data.
|
||||
--gen-field-name Specify field name for --igen. Defaults to "i".
|
||||
--gen-start Specify start value for --igen. Defaults to 1.
|
||||
|
|
@ -401,6 +414,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
--iasv or --iasvlite Use ASV format for input data.
|
||||
--icsv Use CSV format for input data.
|
||||
--icsvlite Use CSV-lite format for input data.
|
||||
--idcf Use Debian control file (DCF) format for input data.
|
||||
--idkvp Use DKVP format for input data.
|
||||
--igen Ignore input files and instead generate sequential
|
||||
numeric input using --gen-field-name, --gen-start,
|
||||
|
|
@ -424,6 +438,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
--oasv or --oasvlite Use ASV format for output data.
|
||||
--ocsv Use CSV format for output data.
|
||||
--ocsvlite Use CSV-lite format for output data.
|
||||
--odcf Use Debian control file (DCF) format for output data.
|
||||
--odkvp Use DKVP format for output data.
|
||||
--ojson Use JSON format for output data.
|
||||
--ojsonl Use JSON Lines format for output data.
|
||||
|
|
@ -470,7 +485,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
1mFORMAT-CONVERSION KEYSTROKE-SAVER FLAGS0m
|
||||
As keystroke-savers for format-conversion you may use the following.
|
||||
The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines,
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively.
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out).
|
||||
|
||||
| In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML |
|
||||
+----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+
|
||||
|
|
@ -837,6 +852,7 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
Format FS PS RS
|
||||
csv "," N/A "\n"
|
||||
csvlite "," N/A "\n"
|
||||
dcf N/A N/A N/A
|
||||
dkvp "," "=" "\n"
|
||||
gen "," N/A "\n"
|
||||
json N/A N/A N/A
|
||||
|
|
@ -3787,5 +3803,5 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
|
||||
https://miller.readthedocs.io
|
||||
|
||||
2026-02-15 4mMILLER24m(1)
|
||||
2026-02-16 4mMILLER24m(1)
|
||||
</pre>
|
||||
|
|
|
|||
|
|
@ -120,6 +120,17 @@
|
|||
| fox jumped | Record 2: "1":"fox", "2":"jumped"
|
||||
+---------------------+
|
||||
|
||||
DCF: Debian control file format
|
||||
+------------+
|
||||
| apple: 1 |
|
||||
| bat: 2 |
|
||||
| cog: 3 |
|
||||
| |
|
||||
| dish: 7 |
|
||||
| egg: 8 |
|
||||
| 3: flint |
|
||||
+------------+
|
||||
|
||||
1mHELP OPTIONS0m
|
||||
Type 'mlr help {topic}' for any of the following:
|
||||
Essentials:
|
||||
|
|
@ -372,6 +383,8 @@
|
|||
--asv or --asvlite Use ASV format for input and output data.
|
||||
--csv or -c or --c2c Use CSV format for input and output data.
|
||||
--csvlite Use CSV-lite format for input and output data.
|
||||
--dcf Use Debian control file (DCF) format for input and
|
||||
output data.
|
||||
--dkvp or --d2d Use DKVP format for input and output data.
|
||||
--gen-field-name Specify field name for --igen. Defaults to "i".
|
||||
--gen-start Specify start value for --igen. Defaults to 1.
|
||||
|
|
@ -380,6 +393,7 @@
|
|||
--iasv or --iasvlite Use ASV format for input data.
|
||||
--icsv Use CSV format for input data.
|
||||
--icsvlite Use CSV-lite format for input data.
|
||||
--idcf Use Debian control file (DCF) format for input data.
|
||||
--idkvp Use DKVP format for input data.
|
||||
--igen Ignore input files and instead generate sequential
|
||||
numeric input using --gen-field-name, --gen-start,
|
||||
|
|
@ -403,6 +417,7 @@
|
|||
--oasv or --oasvlite Use ASV format for output data.
|
||||
--ocsv Use CSV format for output data.
|
||||
--ocsvlite Use CSV-lite format for output data.
|
||||
--odcf Use Debian control file (DCF) format for output data.
|
||||
--odkvp Use DKVP format for output data.
|
||||
--ojson Use JSON format for output data.
|
||||
--ojsonl Use JSON Lines format for output data.
|
||||
|
|
@ -449,7 +464,7 @@
|
|||
1mFORMAT-CONVERSION KEYSTROKE-SAVER FLAGS0m
|
||||
As keystroke-savers for format-conversion you may use the following.
|
||||
The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines,
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively.
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out).
|
||||
|
||||
| In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML |
|
||||
+----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+
|
||||
|
|
@ -816,6 +831,7 @@
|
|||
Format FS PS RS
|
||||
csv "," N/A "\n"
|
||||
csvlite "," N/A "\n"
|
||||
dcf N/A N/A N/A
|
||||
dkvp "," "=" "\n"
|
||||
gen "," N/A "\n"
|
||||
json N/A N/A N/A
|
||||
|
|
@ -3766,4 +3782,4 @@
|
|||
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
|
||||
https://miller.readthedocs.io
|
||||
|
||||
2026-02-15 4mMILLER24m(1)
|
||||
2026-02-16 4mMILLER24m(1)
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ for section_name in section_names
|
|||
# The markdown in this section looks a lot better when hand-crafted (thanks Nikos!).
|
||||
puts <<EOF
|
||||
The letters `c`, `t`, `j`, `l`, `d`, `n`, `x`, `p`, `m`, and `y` refer to formats CSV, TSV, JSON, JSON Lines,
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. Note that markdown format is available for
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use `--dcf` for DCF in and out). Note that markdown format is available for
|
||||
output only.
|
||||
|
||||
| In \\ out | **CSV** | **TSV** | **JSON** | **JSONL** | **DKVP** | **NIDX** | **XTAB** | **PPRINT** | **Markdown** | **YAML** |
|
||||
|
|
|
|||
|
|
@ -47,7 +47,7 @@ See also [Differences from other programming languages](reference-dsl-difference
|
|||
|
||||
Miller's input and output are all text-oriented: all the
|
||||
[file formats supported by Miller](file-formats.md) are human-readable text,
|
||||
such as CSV, TSV, and JSON; binary formats such as
|
||||
such as CSV, TSV, JSON, and DCF; binary formats such as
|
||||
[BSON](https://bsonspec.org/) and [Parquet](https://parquet.apache.org/) are
|
||||
not supported (as of mid-2021). In this sense, everything is a string in and out of
|
||||
Miller -- be it in data files, or in DSL expressions you key in.
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ See also [Differences from other programming languages](reference-dsl-difference
|
|||
|
||||
Miller's input and output are all text-oriented: all the
|
||||
[file formats supported by Miller](file-formats.md) are human-readable text,
|
||||
such as CSV, TSV, and JSON; binary formats such as
|
||||
such as CSV, TSV, JSON, and DCF; binary formats such as
|
||||
[BSON](https://bsonspec.org/) and [Parquet](https://parquet.apache.org/) are
|
||||
not supported (as of mid-2021). In this sense, everything is a string in and out of
|
||||
Miller -- be it in data files, or in DSL expressions you key in.
|
||||
|
|
|
|||
|
|
@ -155,6 +155,7 @@ are overridden in all cases by setting output format to `format2`.
|
|||
* `--asv or --asvlite`: Use ASV format for input and output data.
|
||||
* `--csv or -c or --c2c`: Use CSV format for input and output data.
|
||||
* `--csvlite`: Use CSV-lite format for input and output data.
|
||||
* `--dcf`: Use Debian control file (DCF) format for input and output data.
|
||||
* `--dkvp or --d2d`: Use DKVP format for input and output data.
|
||||
* `--gen-field-name`: Specify field name for --igen. Defaults to "i".
|
||||
* `--gen-start`: Specify start value for --igen. Defaults to 1.
|
||||
|
|
@ -163,6 +164,7 @@ are overridden in all cases by setting output format to `format2`.
|
|||
* `--iasv or --iasvlite`: Use ASV format for input data.
|
||||
* `--icsv`: Use CSV format for input data.
|
||||
* `--icsvlite`: Use CSV-lite format for input data.
|
||||
* `--idcf`: Use Debian control file (DCF) format for input data.
|
||||
* `--idkvp`: Use DKVP format for input data.
|
||||
* `--igen`: Ignore input files and instead generate sequential numeric input using --gen-field-name, --gen-start, --gen-step, and --gen-stop values. See also the seqgen verb, which is more useful/intuitive.
|
||||
* `--ijson`: Use JSON format for input data.
|
||||
|
|
@ -182,6 +184,7 @@ are overridden in all cases by setting output format to `format2`.
|
|||
* `--oasv or --oasvlite`: Use ASV format for output data.
|
||||
* `--ocsv`: Use CSV format for output data.
|
||||
* `--ocsvlite`: Use CSV-lite format for output data.
|
||||
* `--odcf`: Use Debian control file (DCF) format for output data.
|
||||
* `--odkvp`: Use DKVP format for output data.
|
||||
* `--ojson`: Use JSON format for output data.
|
||||
* `--ojsonl`: Use JSON Lines format for output data.
|
||||
|
|
@ -219,7 +222,7 @@ See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatt
|
|||
## Format-conversion keystroke-saver flags
|
||||
|
||||
The letters `c`, `t`, `j`, `l`, `d`, `n`, `x`, `p`, `m`, and `y` refer to formats CSV, TSV, JSON, JSON Lines,
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. Note that markdown format is available for
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use `--dcf` for DCF in and out). Note that markdown format is available for
|
||||
output only.
|
||||
|
||||
| In \ out | **CSV** | **TSV** | **JSON** | **JSONL** | **DKVP** | **NIDX** | **XTAB** | **PPRINT** | **Markdown** | **YAML** |
|
||||
|
|
@ -494,6 +497,7 @@ Notes about all other separators:
|
|||
Format FS PS RS
|
||||
csv "," N/A "\n"
|
||||
csvlite "," N/A "\n"
|
||||
dcf N/A N/A N/A
|
||||
dkvp "," "=" "\n"
|
||||
gen "," N/A "\n"
|
||||
json N/A N/A N/A
|
||||
|
|
|
|||
|
|
@ -276,6 +276,7 @@ Notes:
|
|||
| [**TSV-lite**](file-formats.md#csvtsvasvusvetc) | Default `\n` * | Default `\t` | None |
|
||||
| [**JSON**](file-formats.md#json) | N/A; records are between `{` and `}` | Always `,`; not alterable | Always `:`; not alterable |
|
||||
| [**YAML**](file-formats.md#yaml) | N/A; documents separated by `---` or single array | N/A; not alterable | Always `:`; not alterable |
|
||||
| [**DCF**](file-formats.md#dcf-debian-control-file) | N/A; paragraphs separated by blank lines | N/A; not alterable | Always `:`; not alterable |
|
||||
| [**DKVP**](file-formats.md#dkvp-key-value-pairs) | Default `\n` | Default `,` | Default `=` |
|
||||
| [**NIDX**](file-formats.md#nidx-index-numbered-toolkit-style) | Default `\n` | Default space | None |
|
||||
| [**XTAB**](file-formats.md#xtab-vertical-tabular) | Not used; records are separated by an extra FS | `\n` * | Default: space with repeats |
|
||||
|
|
|
|||
|
|
@ -166,6 +166,7 @@ Notes:
|
|||
| [**TSV-lite**](file-formats.md#csvtsvasvusvetc) | Default `\n` * | Default `\t` | None |
|
||||
| [**JSON**](file-formats.md#json) | N/A; records are between `{` and `}` | Always `,`; not alterable | Always `:`; not alterable |
|
||||
| [**YAML**](file-formats.md#yaml) | N/A; documents separated by `---` or single array | N/A; not alterable | Always `:`; not alterable |
|
||||
| [**DCF**](file-formats.md#dcf-debian-control-file) | N/A; paragraphs separated by blank lines | N/A; not alterable | Always `:`; not alterable |
|
||||
| [**DKVP**](file-formats.md#dkvp-key-value-pairs) | Default `\n` | Default `,` | Default `=` |
|
||||
| [**NIDX**](file-formats.md#nidx-index-numbered-toolkit-style) | Default `\n` | Default space | None |
|
||||
| [**XTAB**](file-formats.md#xtab-vertical-tabular) | Not used; records are separated by an extra FS | `\n` * | Default: space with repeats |
|
||||
|
|
|
|||
3
go.mod
3
go.mod
|
|
@ -40,6 +40,9 @@ require (
|
|||
github.com/google/pprof v0.0.0-20211214055906-6f57359322fd // indirect
|
||||
github.com/kshedden/dstream v0.0.0-20190512025041-c4c410631beb // indirect
|
||||
github.com/pmezard/go-difflib v1.0.0 // indirect
|
||||
golang.org/x/crypto v0.40.0 // indirect
|
||||
golang.org/x/tools v0.41.0 // indirect
|
||||
gonum.org/v1/gonum v0.16.0 // indirect
|
||||
pault.ag/go/debian v0.19.0 // indirect
|
||||
pault.ag/go/topsort v0.1.1 // indirect
|
||||
)
|
||||
|
|
|
|||
6
go.sum
6
go.sum
|
|
@ -41,6 +41,8 @@ github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
|
|||
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
golang.org/x/crypto v0.40.0 h1:r4x+VvoG5Fm+eJcxMaY8CQM7Lb0l1lsmjGBQ6s8BfKM=
|
||||
golang.org/x/crypto v0.40.0/go.mod h1:Qr1vMER5WyS2dfPHAlsOj01wgLbsyWtFn/aY+5+ZdxY=
|
||||
golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
|
||||
|
|
@ -58,3 +60,7 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8
|
|||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
pault.ag/go/debian v0.19.0 h1:RUxCjScMbnlqFH5I+qsmyjZH8fXXtQ05rlkMJop3tjo=
|
||||
pault.ag/go/debian v0.19.0/go.mod h1:1LMojDAazlJ7cA5Ne6H2ZHD4hh3o8NRiW+MpvQRji2o=
|
||||
pault.ag/go/topsort v0.1.1 h1:L0QnhUly6LmTv0e3DEzbN2q6/FGgAcQvaEw65S53Bg4=
|
||||
pault.ag/go/topsort v0.1.1/go.mod h1:r1kc/L0/FZ3HhjezBIPaNVhkqv8L0UJ9bxRuHRVZ0q4=
|
||||
|
|
|
|||
|
|
@ -120,6 +120,17 @@
|
|||
| fox jumped | Record 2: "1":"fox", "2":"jumped"
|
||||
+---------------------+
|
||||
|
||||
DCF: Debian control file format
|
||||
+------------+
|
||||
| apple: 1 |
|
||||
| bat: 2 |
|
||||
| cog: 3 |
|
||||
| |
|
||||
| dish: 7 |
|
||||
| egg: 8 |
|
||||
| 3: flint |
|
||||
+------------+
|
||||
|
||||
1mHELP OPTIONS0m
|
||||
Type 'mlr help {topic}' for any of the following:
|
||||
Essentials:
|
||||
|
|
@ -372,6 +383,8 @@
|
|||
--asv or --asvlite Use ASV format for input and output data.
|
||||
--csv or -c or --c2c Use CSV format for input and output data.
|
||||
--csvlite Use CSV-lite format for input and output data.
|
||||
--dcf Use Debian control file (DCF) format for input and
|
||||
output data.
|
||||
--dkvp or --d2d Use DKVP format for input and output data.
|
||||
--gen-field-name Specify field name for --igen. Defaults to "i".
|
||||
--gen-start Specify start value for --igen. Defaults to 1.
|
||||
|
|
@ -380,6 +393,7 @@
|
|||
--iasv or --iasvlite Use ASV format for input data.
|
||||
--icsv Use CSV format for input data.
|
||||
--icsvlite Use CSV-lite format for input data.
|
||||
--idcf Use Debian control file (DCF) format for input data.
|
||||
--idkvp Use DKVP format for input data.
|
||||
--igen Ignore input files and instead generate sequential
|
||||
numeric input using --gen-field-name, --gen-start,
|
||||
|
|
@ -403,6 +417,7 @@
|
|||
--oasv or --oasvlite Use ASV format for output data.
|
||||
--ocsv Use CSV format for output data.
|
||||
--ocsvlite Use CSV-lite format for output data.
|
||||
--odcf Use Debian control file (DCF) format for output data.
|
||||
--odkvp Use DKVP format for output data.
|
||||
--ojson Use JSON format for output data.
|
||||
--ojsonl Use JSON Lines format for output data.
|
||||
|
|
@ -449,7 +464,7 @@
|
|||
1mFORMAT-CONVERSION KEYSTROKE-SAVER FLAGS0m
|
||||
As keystroke-savers for format-conversion you may use the following.
|
||||
The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines,
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively.
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out).
|
||||
|
||||
| In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML |
|
||||
+----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+
|
||||
|
|
@ -816,6 +831,7 @@
|
|||
Format FS PS RS
|
||||
csv "," N/A "\n"
|
||||
csvlite "," N/A "\n"
|
||||
dcf N/A N/A N/A
|
||||
dkvp "," "=" "\n"
|
||||
gen "," N/A "\n"
|
||||
json N/A N/A N/A
|
||||
|
|
@ -3766,4 +3782,4 @@
|
|||
MIME Type for Comma-Separated Values (CSV) Files, the Miller docsite
|
||||
https://miller.readthedocs.io
|
||||
|
||||
2026-02-15 4mMILLER24m(1)
|
||||
2026-02-16 4mMILLER24m(1)
|
||||
|
|
|
|||
22
man/mlr.1
22
man/mlr.1
|
|
@ -2,12 +2,12 @@
|
|||
.\" Title: mlr
|
||||
.\" Author: [see the "AUTHOR" section]
|
||||
.\" Generator: ./mkman.rb
|
||||
.\" Date: 2026-02-15
|
||||
.\" Date: 2026-02-16
|
||||
.\" Manual: \ \&
|
||||
.\" Source: \ \&
|
||||
.\" Language: English
|
||||
.\"
|
||||
.TH "MILLER" "1" "2026-02-15" "\ \&" "\ \&"
|
||||
.TH "MILLER" "1" "2026-02-16" "\ \&" "\ \&"
|
||||
.\" -----------------------------------------------------------------
|
||||
.\" * Portability definitions
|
||||
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
@ -150,6 +150,17 @@ NIDX: implicitly numerically indexed (Unix-toolkit style)
|
|||
| the quick brown | Record 1: "1":"the", "2":"quick", "3":"brown"
|
||||
| fox jumped | Record 2: "1":"fox", "2":"jumped"
|
||||
+---------------------+
|
||||
|
||||
DCF: Debian control file format
|
||||
+------------+
|
||||
| apple: 1 |
|
||||
| bat: 2 |
|
||||
| cog: 3 |
|
||||
| |
|
||||
| dish: 7 |
|
||||
| egg: 8 |
|
||||
| 3: flint |
|
||||
+------------+
|
||||
.fi
|
||||
.if n \{\
|
||||
.RE
|
||||
|
|
@ -461,6 +472,8 @@ are overridden in all cases by setting output format to `format2`.
|
|||
--asv or --asvlite Use ASV format for input and output data.
|
||||
--csv or -c or --c2c Use CSV format for input and output data.
|
||||
--csvlite Use CSV-lite format for input and output data.
|
||||
--dcf Use Debian control file (DCF) format for input and
|
||||
output data.
|
||||
--dkvp or --d2d Use DKVP format for input and output data.
|
||||
--gen-field-name Specify field name for --igen. Defaults to "i".
|
||||
--gen-start Specify start value for --igen. Defaults to 1.
|
||||
|
|
@ -469,6 +482,7 @@ are overridden in all cases by setting output format to `format2`.
|
|||
--iasv or --iasvlite Use ASV format for input data.
|
||||
--icsv Use CSV format for input data.
|
||||
--icsvlite Use CSV-lite format for input data.
|
||||
--idcf Use Debian control file (DCF) format for input data.
|
||||
--idkvp Use DKVP format for input data.
|
||||
--igen Ignore input files and instead generate sequential
|
||||
numeric input using --gen-field-name, --gen-start,
|
||||
|
|
@ -492,6 +506,7 @@ are overridden in all cases by setting output format to `format2`.
|
|||
--oasv or --oasvlite Use ASV format for output data.
|
||||
--ocsv Use CSV format for output data.
|
||||
--ocsvlite Use CSV-lite format for output data.
|
||||
--odcf Use Debian control file (DCF) format for output data.
|
||||
--odkvp Use DKVP format for output data.
|
||||
--ojson Use JSON format for output data.
|
||||
--ojsonl Use JSON Lines format for output data.
|
||||
|
|
@ -554,7 +569,7 @@ See the flatten/unflatten doc page https://miller.readthedocs.io/en/latest/flatt
|
|||
.nf
|
||||
As keystroke-savers for format-conversion you may use the following.
|
||||
The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines,
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively.
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out).
|
||||
|
||||
| In\eout | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML |
|
||||
+----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+
|
||||
|
|
@ -977,6 +992,7 @@ Notes about all other separators:
|
|||
Format FS PS RS
|
||||
csv "," N/A "\en"
|
||||
csvlite "," N/A "\en"
|
||||
dcf N/A N/A N/A
|
||||
dkvp "," "=" "\en"
|
||||
gen "," N/A "\en"
|
||||
json N/A N/A N/A
|
||||
|
|
|
|||
|
|
@ -61,7 +61,8 @@ package cli
|
|||
func DecideFinalFlatten(writerOptions *TWriterOptions) bool {
|
||||
ofmt := writerOptions.OutputFileFormat
|
||||
if writerOptions.AutoFlatten {
|
||||
if ofmt != "json" {
|
||||
// Preserve nested/array structure for formats that support it.
|
||||
if ofmt != "json" && ofmt != "jsonl" && ofmt != "dcf" {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -787,6 +787,15 @@ var FileFormatFlagSection = FlagSection{
|
|||
},
|
||||
},
|
||||
|
||||
{
|
||||
name: "--idcf",
|
||||
help: "Use Debian control file (DCF) format for input data.",
|
||||
parser: func(args []string, argc int, pargi *int, options *TOptions) {
|
||||
options.ReaderOptions.InputFileFormat = "dcf"
|
||||
*pargi += 1
|
||||
},
|
||||
},
|
||||
|
||||
{
|
||||
name: "--inidx",
|
||||
help: "Use NIDX format for input data.",
|
||||
|
|
@ -1021,6 +1030,15 @@ var FileFormatFlagSection = FlagSection{
|
|||
},
|
||||
},
|
||||
|
||||
{
|
||||
name: "--odcf",
|
||||
help: "Use Debian control file (DCF) format for output data.",
|
||||
parser: func(args []string, argc int, pargi *int, options *TOptions) {
|
||||
options.WriterOptions.OutputFileFormat = "dcf"
|
||||
*pargi += 1
|
||||
},
|
||||
},
|
||||
|
||||
{
|
||||
name: "--onidx",
|
||||
help: "Use NIDX format for output data.",
|
||||
|
|
@ -1200,6 +1218,16 @@ var FileFormatFlagSection = FlagSection{
|
|||
},
|
||||
},
|
||||
|
||||
{
|
||||
name: "--dcf",
|
||||
help: "Use Debian control file (DCF) format for input and output data.",
|
||||
parser: func(args []string, argc int, pargi *int, options *TOptions) {
|
||||
options.ReaderOptions.InputFileFormat = "dcf"
|
||||
options.WriterOptions.OutputFileFormat = "dcf"
|
||||
*pargi += 1
|
||||
},
|
||||
},
|
||||
|
||||
{
|
||||
name: "--nidx",
|
||||
help: "Use NIDX format for input and output data.",
|
||||
|
|
@ -1252,7 +1280,7 @@ var FileFormatFlagSection = FlagSection{
|
|||
func FormatConversionKeystrokeSaverPrintInfo() {
|
||||
fmt.Println(`As keystroke-savers for format-conversion you may use the following.
|
||||
The letters c, t, j, l, d, n, x, p, m, and y refer to formats CSV, TSV, JSON, JSON Lines,
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively.
|
||||
DKVP, NIDX, XTAB, PPRINT, markdown, and YAML, respectively. DCF is also supported (use --dcf for DCF in and out).
|
||||
|
||||
| In\out | CSV | TSV | JSON | JSONL | DKVP | NIDX | XTAB | PPRINT | Markdown | YAML |
|
||||
+----------+----------+----------+----------+-------+-------+-------+-------+--------+----------+--------+
|
||||
|
|
|
|||
|
|
@ -88,6 +88,7 @@ var defaultFSes = map[string]string{
|
|||
"dkvp": ",",
|
||||
"json": "N/A", // not alterable; not parameterizable in JSON format
|
||||
"yaml": "N/A",
|
||||
"dcf": "N/A",
|
||||
"nidx": " ",
|
||||
"markdown": " ",
|
||||
"pprint": " ",
|
||||
|
|
@ -102,6 +103,7 @@ var defaultPSes = map[string]string{
|
|||
"dkvp": "=",
|
||||
"json": "N/A", // not alterable; not parameterizable in JSON format
|
||||
"yaml": "N/A",
|
||||
"dcf": "N/A",
|
||||
"markdown": "N/A",
|
||||
"nidx": "N/A",
|
||||
"pprint": "N/A",
|
||||
|
|
@ -116,6 +118,7 @@ var defaultRSes = map[string]string{
|
|||
"dkvp": "\n",
|
||||
"json": "N/A", // not alterable; not parameterizable in JSON format
|
||||
"yaml": "N/A",
|
||||
"dcf": "N/A",
|
||||
"markdown": "\n",
|
||||
"nidx": "\n",
|
||||
"pprint": "\n",
|
||||
|
|
@ -130,6 +133,7 @@ var defaultAllowRepeatIFSes = map[string]bool{
|
|||
"dkvp": false,
|
||||
"json": false,
|
||||
"yaml": false,
|
||||
"dcf": false,
|
||||
"markdown": false,
|
||||
"nidx": false,
|
||||
"pprint": true,
|
||||
|
|
|
|||
198
pkg/input/record_reader_dcf.go
Normal file
198
pkg/input/record_reader_dcf.go
Normal file
|
|
@ -0,0 +1,198 @@
|
|||
package input
|
||||
|
||||
import (
|
||||
"io"
|
||||
"strings"
|
||||
|
||||
"pault.ag/go/debian/control"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/cli"
|
||||
"github.com/johnkerl/miller/v6/pkg/lib"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/types"
|
||||
)
|
||||
|
||||
// DCF list-valued field names (comma-separated in the format). These are
|
||||
// exposed as Miller arrays; all other fields remain strings.
|
||||
var dcfListFieldNames = map[string]bool{
|
||||
"Depends": true,
|
||||
"Pre-Depends": true,
|
||||
"Recommends": true,
|
||||
"Suggests": true,
|
||||
"Enhances": true,
|
||||
"Breaks": true,
|
||||
"Conflicts": true,
|
||||
"Replaces": true,
|
||||
"Build-Depends": true,
|
||||
"Build-Depends-Indep": true,
|
||||
"Build-Conflicts": true,
|
||||
"Build-Conflicts-Indep": true,
|
||||
"Built-Using": true,
|
||||
}
|
||||
|
||||
type RecordReaderDCF struct {
|
||||
readerOptions *cli.TReaderOptions
|
||||
recordsPerBatch int64
|
||||
}
|
||||
|
||||
func NewRecordReaderDCF(
|
||||
readerOptions *cli.TReaderOptions,
|
||||
recordsPerBatch int64,
|
||||
) (*RecordReaderDCF, error) {
|
||||
return &RecordReaderDCF{
|
||||
readerOptions: readerOptions,
|
||||
recordsPerBatch: recordsPerBatch,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (reader *RecordReaderDCF) Read(
|
||||
filenames []string,
|
||||
context types.Context,
|
||||
readerChannel chan<- []*types.RecordAndContext,
|
||||
errorChannel chan error,
|
||||
downstreamDoneChannel <-chan bool,
|
||||
) {
|
||||
if filenames != nil {
|
||||
if len(filenames) == 0 {
|
||||
handle, err := lib.OpenStdin(
|
||||
reader.readerOptions.Prepipe,
|
||||
reader.readerOptions.PrepipeIsRaw,
|
||||
reader.readerOptions.FileInputEncoding,
|
||||
)
|
||||
if err != nil {
|
||||
errorChannel <- err
|
||||
} else {
|
||||
reader.processHandle(handle, "(stdin)", &context, readerChannel, errorChannel, downstreamDoneChannel)
|
||||
}
|
||||
} else {
|
||||
for _, filename := range filenames {
|
||||
handle, err := lib.OpenFileForRead(
|
||||
filename,
|
||||
reader.readerOptions.Prepipe,
|
||||
reader.readerOptions.PrepipeIsRaw,
|
||||
reader.readerOptions.FileInputEncoding,
|
||||
)
|
||||
if err != nil {
|
||||
errorChannel <- err
|
||||
} else {
|
||||
reader.processHandle(handle, filename, &context, readerChannel, errorChannel, downstreamDoneChannel)
|
||||
handle.Close()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
readerChannel <- types.NewEndOfStreamMarkerList(&context)
|
||||
}
|
||||
|
||||
func (reader *RecordReaderDCF) processHandle(
|
||||
handle io.Reader,
|
||||
filename string,
|
||||
context *types.Context,
|
||||
readerChannel chan<- []*types.RecordAndContext,
|
||||
errorChannel chan error,
|
||||
downstreamDoneChannel <-chan bool,
|
||||
) {
|
||||
context.UpdateForStartOfFile(filename)
|
||||
recordsPerBatch := reader.recordsPerBatch
|
||||
|
||||
pr, err := control.NewParagraphReader(handle, nil)
|
||||
if err != nil {
|
||||
errorChannel <- err
|
||||
return
|
||||
}
|
||||
|
||||
recordsAndContexts := make([]*types.RecordAndContext, 0, recordsPerBatch)
|
||||
i := int64(0)
|
||||
|
||||
for {
|
||||
i++
|
||||
if i%recordsPerBatch == 0 {
|
||||
select {
|
||||
case <-downstreamDoneChannel:
|
||||
goto flush
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
para, err := pr.Next()
|
||||
if err != nil {
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
errorChannel <- err
|
||||
return
|
||||
}
|
||||
if para == nil {
|
||||
break
|
||||
}
|
||||
|
||||
rec := dcfParagraphToRecord(para, reader.readerOptions.DedupeFieldNames)
|
||||
if rec == nil {
|
||||
// dedupe or other error already sent
|
||||
return
|
||||
}
|
||||
|
||||
context.UpdateForInputRecord()
|
||||
recordsAndContexts = append(recordsAndContexts, types.NewRecordAndContext(rec, context))
|
||||
if int64(len(recordsAndContexts)) >= recordsPerBatch {
|
||||
readerChannel <- recordsAndContexts
|
||||
recordsAndContexts = make([]*types.RecordAndContext, 0, recordsPerBatch)
|
||||
}
|
||||
}
|
||||
|
||||
flush:
|
||||
if len(recordsAndContexts) > 0 {
|
||||
readerChannel <- recordsAndContexts
|
||||
}
|
||||
}
|
||||
|
||||
// splitCommaList splits a DCF comma-separated value and returns Miller Mlrval
|
||||
// array elements (trimmed, skipping empty).
|
||||
func splitCommaList(s string) []*mlrval.Mlrval {
|
||||
parts := strings.Split(s, ",")
|
||||
out := make([]*mlrval.Mlrval, 0, len(parts))
|
||||
for _, p := range parts {
|
||||
p = strings.TrimSpace(p)
|
||||
if p == "" {
|
||||
continue
|
||||
}
|
||||
out = append(out, mlrval.FromString(p))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// dcfParagraphToRecord converts a control paragraph to a Miller record. List
|
||||
// fields (Depends, Build-Depends, etc.) become arrays; all other fields stay
|
||||
// strings. Uses para.Order to preserve field order. Returns nil on error (caller
|
||||
// must have sent on errorChannel).
|
||||
func dcfParagraphToRecord(para *control.Paragraph, dedupeFieldNames bool) *mlrval.Mlrmap {
|
||||
rec := mlrval.NewMlrmapAsRecord()
|
||||
|
||||
// Preserve order from the DCF file when present.
|
||||
keys := para.Order
|
||||
if len(keys) == 0 {
|
||||
for k := range para.Values {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
}
|
||||
|
||||
for _, k := range keys {
|
||||
v, ok := para.Values[k]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
var mv *mlrval.Mlrval
|
||||
if dcfListFieldNames[k] {
|
||||
mv = mlrval.FromArray(splitCommaList(v))
|
||||
} else {
|
||||
mv = mlrval.FromString(v)
|
||||
}
|
||||
|
||||
if _, err := rec.PutReferenceMaybeDedupe(k, mv, dedupeFieldNames); err != nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
return rec
|
||||
}
|
||||
|
|
@ -30,6 +30,8 @@ func Create(readerOptions *cli.TReaderOptions, recordsPerBatch int64) (IRecordRe
|
|||
return NewRecordReaderTSV(readerOptions, recordsPerBatch)
|
||||
case "xtab":
|
||||
return NewRecordReaderXTAB(readerOptions, recordsPerBatch)
|
||||
case "dcf":
|
||||
return NewRecordReaderDCF(readerOptions, recordsPerBatch)
|
||||
case "gen":
|
||||
return NewPseudoReaderGen(readerOptions, recordsPerBatch)
|
||||
default:
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
// Wrapper for os.Open which maps string filename to *os.File, which in turn
|
||||
// implements io.ReadCloser, and optional in turn wrapping that in a
|
||||
// gzip/zlib/bunzip2 reader. Shared across record-readers for all the various
|
||||
// input-file formats (CSV, JSON, XTAB, DKVP, NIDX, PPRINT) which Miller
|
||||
// input-file formats (CSV, JSON, XTAB, DKVP, NIDX, PPRINT, DCF) which Miller
|
||||
// supports.
|
||||
//
|
||||
// There are two ways of handling compressed data in the Miller Go port:
|
||||
|
|
|
|||
85
pkg/output/record_writer_dcf.go
Normal file
85
pkg/output/record_writer_dcf.go
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
package output
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"strings"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/cli"
|
||||
"github.com/johnkerl/miller/v6/pkg/colorizer"
|
||||
"github.com/johnkerl/miller/v6/pkg/mlrval"
|
||||
"github.com/johnkerl/miller/v6/pkg/types"
|
||||
)
|
||||
|
||||
type RecordWriterDCF struct {
|
||||
writerOptions *cli.TWriterOptions
|
||||
}
|
||||
|
||||
func NewRecordWriterDCF(writerOptions *cli.TWriterOptions) (*RecordWriterDCF, error) {
|
||||
return &RecordWriterDCF{
|
||||
writerOptions: writerOptions,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (writer *RecordWriterDCF) Write(
|
||||
outrec *mlrval.Mlrmap,
|
||||
_ *types.Context,
|
||||
bufferedOutputStream *bufio.Writer,
|
||||
outputIsStdout bool,
|
||||
) error {
|
||||
if outrec == nil {
|
||||
return nil
|
||||
}
|
||||
if outrec.IsEmpty() {
|
||||
bufferedOutputStream.WriteString("\n")
|
||||
return nil
|
||||
}
|
||||
|
||||
for pe := outrec.Head; pe != nil; pe = pe.Next {
|
||||
valueStr := dcfValueString(pe.Value)
|
||||
keyStr := colorizer.MaybeColorizeKey(pe.Key, outputIsStdout)
|
||||
valueStr = colorizer.MaybeColorizeValue(valueStr, outputIsStdout)
|
||||
writeDCFField(bufferedOutputStream, keyStr, valueStr)
|
||||
}
|
||||
bufferedOutputStream.WriteString("\n")
|
||||
return nil
|
||||
}
|
||||
|
||||
// dcfValueString returns the string form of a field value for DCF output.
|
||||
// Arrays are joined with ", " to match list fields (Depends, etc.).
|
||||
func dcfValueString(mv *mlrval.Mlrval) string {
|
||||
if mv == nil {
|
||||
return ""
|
||||
}
|
||||
if mv.IsArray() {
|
||||
arr := mv.GetArray()
|
||||
if arr == nil || len(arr) == 0 {
|
||||
return ""
|
||||
}
|
||||
parts := make([]string, 0, len(arr))
|
||||
for _, el := range arr {
|
||||
if el != nil {
|
||||
parts = append(parts, el.String())
|
||||
}
|
||||
}
|
||||
return strings.Join(parts, ", ")
|
||||
}
|
||||
return mv.String()
|
||||
}
|
||||
|
||||
// writeDCFField writes one "Key: value" line, folding on newlines per DCF:
|
||||
// continuation lines start with a single space.
|
||||
func writeDCFField(b *bufio.Writer, key, value string) {
|
||||
lines := strings.Split(value, "\n")
|
||||
for i, line := range lines {
|
||||
if i == 0 {
|
||||
b.WriteString(key)
|
||||
b.WriteString(": ")
|
||||
b.WriteString(line)
|
||||
b.WriteString("\n")
|
||||
} else {
|
||||
b.WriteString(" ")
|
||||
b.WriteString(line)
|
||||
b.WriteString("\n")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -20,6 +20,8 @@ func Create(writerOptions *cli.TWriterOptions) (IRecordWriter, error) {
|
|||
return NewRecordWriterJSONLines(writerOptions)
|
||||
case "yaml":
|
||||
return NewRecordWriterYAML(writerOptions)
|
||||
case "dcf":
|
||||
return NewRecordWriterDCF(writerOptions)
|
||||
case "md":
|
||||
return NewRecordWriterMarkdown(writerOptions)
|
||||
case "markdown":
|
||||
|
|
|
|||
|
|
@ -447,6 +447,17 @@ NIDX: implicitly numerically indexed (Unix-toolkit style)
|
|||
| the quick brown | Record 1: "1":"the", "2":"quick", "3":"brown"
|
||||
| fox jumped | Record 2: "1":"fox", "2":"jumped"
|
||||
+---------------------+
|
||||
|
||||
DCF: Debian control file format
|
||||
+------------+
|
||||
| apple: 1 |
|
||||
| bat: 2 |
|
||||
| cog: 3 |
|
||||
| |
|
||||
| dish: 7 |
|
||||
| egg: 8 |
|
||||
| 3: flint |
|
||||
+------------+
|
||||
`)
|
||||
}
|
||||
|
||||
|
|
|
|||
1
test/cases/io-dcf/0001/cmd
Normal file
1
test/cases/io-dcf/0001/cmd
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr --itsv --rs lf --odcf cat test/input/simple.tsv
|
||||
0
test/cases/io-dcf/0001/experr
Normal file
0
test/cases/io-dcf/0001/experr
Normal file
30
test/cases/io-dcf/0001/expout
Normal file
30
test/cases/io-dcf/0001/expout
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
a: pan
|
||||
b: pan
|
||||
i: 1
|
||||
x: 2
|
||||
y: 0.98994500
|
||||
|
||||
a: eks
|
||||
b: pan
|
||||
i: 2
|
||||
x: 1
|
||||
y: 0.77515900
|
||||
|
||||
a: wye
|
||||
b: wye
|
||||
i: 3
|
||||
x: 1
|
||||
y: 0.76164200
|
||||
|
||||
a: eks
|
||||
b: wye
|
||||
i: 4
|
||||
x: 5
|
||||
y: 0.32293400
|
||||
|
||||
a: wye
|
||||
b: pan
|
||||
i: 5
|
||||
x: 5
|
||||
y: 0.44828300
|
||||
|
||||
1
test/cases/io-dcf/0002/cmd
Normal file
1
test/cases/io-dcf/0002/cmd
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr --icsv --odcf cat ${CASEDIR}/input
|
||||
0
test/cases/io-dcf/0002/experr
Normal file
0
test/cases/io-dcf/0002/experr
Normal file
4
test/cases/io-dcf/0002/expout
Normal file
4
test/cases/io-dcf/0002/expout
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
a: 1
|
||||
b: 2|3
|
||||
c: 4
|
||||
|
||||
2
test/cases/io-dcf/0002/input
Normal file
2
test/cases/io-dcf/0002/input
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
a,b,c
|
||||
1,2|3,4
|
||||
1
test/cases/io-dcf/0003/cmd
Normal file
1
test/cases/io-dcf/0003/cmd
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr --idcf --odcf cat test/input/test.dcf
|
||||
0
test/cases/io-dcf/0003/experr
Normal file
0
test/cases/io-dcf/0003/experr
Normal file
10
test/cases/io-dcf/0003/expout
Normal file
10
test/cases/io-dcf/0003/expout
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
Package: foo
|
||||
Version: 1.0
|
||||
Depends: libc6 (>= 2.0), libfoo (>= 1.2)
|
||||
Description: A test package.
|
||||
|
||||
Package: bar
|
||||
Version: 2.0
|
||||
Recommends: foo
|
||||
Description: Another package.
|
||||
|
||||
1
test/cases/io-dcf/dcf-input/cmd
Normal file
1
test/cases/io-dcf/dcf-input/cmd
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr --idcf --ojson cat test/input/test.dcf
|
||||
0
test/cases/io-dcf/dcf-input/experr
Normal file
0
test/cases/io-dcf/dcf-input/experr
Normal file
14
test/cases/io-dcf/dcf-input/expout
Normal file
14
test/cases/io-dcf/dcf-input/expout
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
[
|
||||
{
|
||||
"Package": "foo",
|
||||
"Version": "1.0",
|
||||
"Depends": ["libc6 (>= 2.0)", "libfoo (>= 1.2)"],
|
||||
"Description": "A test package."
|
||||
},
|
||||
{
|
||||
"Package": "bar",
|
||||
"Version": "2.0",
|
||||
"Recommends": ["foo"],
|
||||
"Description": "Another package."
|
||||
}
|
||||
]
|
||||
9
test/input/test.dcf
Normal file
9
test/input/test.dcf
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
Package: foo
|
||||
Version: 1.0
|
||||
Depends: libc6 (>= 2.0), libfoo (>= 1.2)
|
||||
Description: A test package.
|
||||
|
||||
Package: bar
|
||||
Version: 2.0
|
||||
Recommends: foo
|
||||
Description: Another package.
|
||||
Loading…
Add table
Add a link
Reference in a new issue