mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-17 16:38:54 +00:00
b7804e0791
Implements the plan in plans/recutils.md, addressing #378. Adds a hand-rolled reader/writer pair (pkg/input/record_reader_rec.go, pkg/output/record_writer_rec.go) supporting recutils' blank-line-separated "Key: value" records, "+"-continuation and backslash-newline continuation, and generic comment handling, with recutils' %rec-descriptor schema layer left unspecialized since Miller has no schema-enforcement concept. Registers the format under the "recutils" name (--irecutils/--orecutils/ --recutils flags, matching the --idcf/--odcf/--dcf pattern) in the reader/writer factories and pkg/cli/separators.go, adds docs and a sample data file, and adds unit tests plus test/cases/io-recutils regression cases covering round-tripping, continuation lines, comments, and the hard-error behavior on malformed input. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
178 lines
8.2 KiB
Markdown
178 lines
8.2 KiB
Markdown
# Separators
|
|
|
|
## Record, field, and pair separators
|
|
|
|
Miller has record separators, field separators, and pair separators. For
|
|
example, given the following [DKVP](file-formats.md#dkvp-key-value-pairs)
|
|
records:
|
|
|
|
GENMD-RUN-COMMAND
|
|
cat data/a.dkvp
|
|
GENMD-EOF
|
|
|
|
* the **record separator** is newline -- it separates records from one another;
|
|
* the **field separator** is `,` -- it separates fields (key-value pairs) from one another;
|
|
* and the **pair separator** is `=` -- it separates the key from the value within each key-value pair.
|
|
|
|
These are the default values, which you can override with flags such as `--ips`
|
|
and `--ops` (below).
|
|
|
|
Not all [file formats](file-formats.md) have all three of these: for example,
|
|
CSV does not have a pair separator, since keys are on the header line and
|
|
values are on each data line.
|
|
|
|
Also, separators are not programmable for all file formats. For example, in
|
|
[JSON objects](file-formats.md#json), the pair separator is `:` and the
|
|
field-separator is `,` -- we write `{"a":1,"b":2,"c":3}` -- but these aren't
|
|
modifiable. If you do `mlr --json --ips : --ips '=' cat myfile.json` then you
|
|
don't get `{"a"=1,"b"=2,"c"=3}`. This is because the pair-separator `:` is
|
|
part of the JSON specification.
|
|
|
|
## Input and output separators
|
|
|
|
Miller lets you use the same separators for input and output (e.g. CSV input,
|
|
CSV output), or, to change them between input and output (e.g. CSV input, JSON
|
|
output), if you wish to transform your data in that way.
|
|
|
|
Miller uses the names `IRS` and `ORS` for the input and output record
|
|
separators, `IFS` and `OFS` for the input and output field separators, and
|
|
`IPS` and `OPS` for input and output pair separators.
|
|
|
|
For example:
|
|
|
|
GENMD-RUN-COMMAND
|
|
cat data/a.dkvp
|
|
GENMD-EOF
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr --ifs , --ofs ';' --ips = --ops : cut -o -f c,a,b data/a.dkvp
|
|
GENMD-EOF
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr --csv head -n 2 example.csv
|
|
GENMD-EOF
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr --csv --ofs pipe head -n 2 example.csv
|
|
GENMD-EOF
|
|
|
|
If your data has non-default separators and you don't want to change those
|
|
between input and output, you can use `--rs`, `--fs`, and `--ps`. Setting `--fs
|
|
:` is the same as setting `--ifs : --ofs :`, but with fewer keystrokes.
|
|
|
|
GENMD-RUN-COMMAND
|
|
cat data/modsep.dkvp
|
|
GENMD-EOF
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr --fs ';' --ps : cut -o -f c,a,b data/modsep.dkvp
|
|
GENMD-EOF
|
|
|
|
## Multi-character separators
|
|
|
|
All separators can be multi-character, except for file formats which don't
|
|
allow parameterization (see below). And for CSV (CSV-lite doesn't have these
|
|
restrictions), IRS must be `\n` and IFS must be a single character.
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr --ifs ';' --ips : --ofs ';;;' --ops := cut -o -f c,a,b data/modsep.dkvp
|
|
GENMD-EOF
|
|
|
|
If your data has field separators which are one or more consecutive spaces, you
|
|
can use `--ifs space --repifs`.
|
|
More generally, the `--repifs` flag means that multiple successive occurrences of the field
|
|
separator count as one. For example, in CSV data we often signify nulls by
|
|
empty strings, e.g. `2,9,,,,,6,5,4`. On the other hand, if the field separator
|
|
is a space, it might be more natural to parse `2 4 5` the same as `2 4 5`:
|
|
`--repifs --ifs ' '` lets this happen. In fact, the `--ipprint` option
|
|
is internally implemented in terms of `--repifs`.
|
|
|
|
For example:
|
|
|
|
GENMD-RUN-COMMAND
|
|
cat data/extra-spaces.txt
|
|
GENMD-EOF
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr --ifs ' ' --repifs --inidx --oxtab cat data/extra-spaces.txt
|
|
GENMD-EOF
|
|
|
|
## Regular-expression separators
|
|
|
|
`IFS` and `IPS` can be regular expressions: use `--ifs-regex` or `--ips-regex` in place of
|
|
`--ifs` or `--ips`, respectively.
|
|
|
|
You can also use either `--ifs space --repifs` or `--ifs-regex '()+'`. (But that gets a little tedious,
|
|
so there are aliases listed below.) Note however that `--ifs space --repifs` is about 3x faster than
|
|
`--ifs-regex '( )+'` -- regular expressions are powerful, but slower.
|
|
|
|
## Aliases
|
|
|
|
Many things we'd like to write as separators need to be escaped from the shell
|
|
-- e.g. `--ifs ';'` or `--ofs '|'`, and so on. You can use the following if you like:
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr help list-separator-aliases
|
|
GENMD-EOF
|
|
|
|
And for `--ifs-regex` and `--ips-regex`:
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr help list-separator-regex-aliases
|
|
GENMD-EOF
|
|
|
|
Note that `spaces`, `tabs`, and `whitespace` already are regexes so you
|
|
shouldn't use `--repifs` with them. (In fact, the `--repifs` flag is ignored
|
|
when `--ifs-regex` is provided.)
|
|
|
|
## Command-line flags
|
|
|
|
Given the above, we now have seen the following flags:
|
|
|
|
GENMD-CARDIFY
|
|
--rs --irs --ors
|
|
--fs --ifs --ofs --repifs --ifs-regex
|
|
--ps --ips --ops --ips-regex
|
|
GENMD-EOF
|
|
|
|
See also the [separator-flags section](reference-main-flag-list.md#separator-flags).
|
|
|
|
## DSL built-in variables
|
|
|
|
Miller exposes for you read-only [built-in variables](reference-dsl-variables.md#built-in-variables) with
|
|
names `IRS`, `ORS`, `IFS`, `OFS`, `IPS`, and `OPS`. Unlike in AWK, you can't set these in begin-blocks --
|
|
their values indicate what you specified at the command line -- so their use is limited.
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr --ifs , --ofs ';' --ips = --ops : --from data/a.dkvp put '$d = ">>>" . IFS . "|||" . OFS . "<<<"'
|
|
GENMD-EOF
|
|
|
|
## Which separators apply to which file formats
|
|
|
|
Notes:
|
|
|
|
* CSV IRS must be newline -- CR/LF line endings are accepted on input. CSV ORS must be either newline (the default) or carriage-return/newline -- e.g. `--ors crlf` or `--ors '\r\n'` for RFC-4180-style line endings on any platform. CSV IFS must be a single character. (CSV-lite does not have these restrictions.)
|
|
* TSV IRS must be newline -- CR/LF line endings are accepted on input. TSV ORS must be either newline (the default) or carriage-return/newline. TSV IFS must be a tab. (TSV-lite does not have these restrictions.)
|
|
* See the [CSV section](file-formats.md#csvtsvasvusvetc) for information about ASV and USV.
|
|
* JSON and YAML: ignore separator flags from the command line.
|
|
* Headerless CSV overlaps quite a bit with NIDX format using comma for IFS. See also the page on [CSV with and without headers](csv-with-and-without-headers.md).
|
|
* For XTAB, the record separator is a repetition of the field separator. For example, if one record has `x=1,y=2` and the next has `x=3,y=4`, and OFS is newline, then output lines are `x 1`, then `y 2`, then an extra newline, then `x 3`, then `y 4`. This means: to customize XTAB, set `OFS` rather than `ORS`.
|
|
|
|
| | **RS** | **FS** | **PS** |
|
|
|------------|---------|---------|----------|
|
|
| [**CSV**](file-formats.md#csvtsvasvusvetc) | Default `\n`; may be set to `\r\n` * | Default `,`; must be single-character | None |
|
|
| [**TSV**](file-formats.md#csvtsvasvusvetc) | Default `\n`; may be set to `\r\n` * | Default `\t`; must be single-character | None |
|
|
| [**CSV-lite**](file-formats.md#csvtsvasvusvetc) | Default `\n` * | Default `,` | None |
|
|
| [**TSV-lite**](file-formats.md#csvtsvasvusvetc) | Default `\n` * | Default `\t` | None |
|
|
| [**JSON**](file-formats.md#json) | N/A; records are between `{` and `}` | Always `,`; not alterable | Always `:`; not alterable |
|
|
| [**YAML**](file-formats.md#yaml) | N/A; documents separated by `---` or single array | N/A; not alterable | Always `:`; not alterable |
|
|
| [**DCF**](file-formats.md#dcf-debian-control-file) | N/A; paragraphs separated by blank lines | N/A; not alterable | Always `:`; not alterable |
|
|
| [**recutils**](file-formats.md#recutils) | N/A; records separated by blank lines | N/A; not alterable | Always `: `; not alterable |
|
|
| [**DKVP**](file-formats.md#dkvp-key-value-pairs) | Default `\n` | Default `,` | Default `=` |
|
|
| [**DKVPX**](file-formats.md#dkvpx-key-value-pairs-with-csv-style-quoting) | Default `\n` | Default `,`; must be single-character for input | Default `=`; must be single-character for input |
|
|
| [**NIDX**](file-formats.md#nidx-index-numbered-toolkit-style) | Default `\n` | Default space | None |
|
|
| [**XTAB**](file-formats.md#xtab-vertical-tabular) | Not used; records are separated by an extra FS | `\n` * | Default: space with repeats |
|
|
| [**PPRINT**](file-formats.md#pprint-pretty-printed-tabular) | Default `\n` * | Space with repeats | None |
|
|
| [**Markdown**](file-formats.md#markdown-tabular) | Always `\n`; not alterable * | One or more spaces, then `|`, then one or more spaces; not alterable | None |
|
|
|
|
\* or `\r\n` on Windows
|