# File formats

Miller handles name-indexed data using several formats: some you probably know
by name, such as CSV, TSV, JSON, and JSON Lines -- and other formats you're likely already
seeing and using in your structured data.

Additionally, Miller gives you the option to include comments within your data.

## Examples

GENMD-RUN-COMMAND
mlr help file-formats
GENMD-EOF

## CSV/TSV/ASV/USV/etc.

When `mlr` is invoked with the `--csv` or `--csvlite` option, key names are found on the first record, and values are taken from subsequent records.  This includes the case of CSV-formatted files.  See [Record Heterogeneity](record-heterogeneity.md) for how Miller handles changes of field names within a single data stream.

Miller has record separator `RS` and field separator `FS`, just as `awk` does. (See also the [separators page](reference-main-separators.md).)

**CSV (comma-separated values):** Miller's `--csv` flag supports [RFC-4180 CSV](https://tools.ietf.org/html/rfc4180).

* This includes CRLF line terminators by default, regardless of platform.
* Any cell containing a comma or a carriage return within it must be double-quoted.

**TSV (tab-separated values):** Miller's `--tsv` supports [IANA TSV](https://www.iana.org/assignments/media-types/text/tab-separated-values).

* `FS` is tab and `RS` is newline (or carriage return + linefeed for Windows).
* On input, if fields have `\r`, `\n`, `\t`, or `\\`, those are decoded as carriage return, newline, tab, and backslash, respectively.
* On output, the reverse is done -- for example, if a field has an embedded newline, that newline is replaced by `\n`.
* A tab within a cell must be encoded as `\t`.
* A carriage return within a cell must be encoded as `\n`.

**ASV (ASCII-separated values):** the flags `--asv`, `--iasv`, `--oasv`, `--asvlite`, `--iasvlite`, and `--oasvlite` are analogous except they use ASCII FS and RS `0x1f` and `0x1e`, respectively.

**USV (Unicode-separated values):** likewise, the flags `--usv`, `--iusv`, `--ousv`, `--usvlite`, `--iusvlite`, and `--ousvlite` use Unicode FS and RS `U+241F` (UTF-8 `0x0xe2909f`) and `U+241E` (UTF-8 `0xe2909e`), respectively.

Here are the differences between CSV and CSV-lite:

* CSV-lite naively splits lines on newline, and fields on comma -- embedded commas and newlines are not escaped in any way.

* CSV supports [RFC-4180](https://tools.ietf.org/html/rfc4180)-style double-quoting, including the ability to have commas and/or LF/CRLF line-endings contained within an input field; CSV-lite does not.

* CSV does not allow heterogeneous data; CSV-lite does (see also [Record Heterogeneity](record-heterogeneity.md)).

* TSV-lite is simply CSV-lite with the field separator set to tab instead of a comma.
In particular, no encoding/decoding of  `\r`, `\n`, `\t`, or `\\` is done.

* CSV-lite allows changing FS and/or RS to any values, perhaps multi-character.

* CSV-lite and TSV-lite handle schema changes ("schema" meaning "ordered list of field names in a given record") by adding a newline and re-emitting the header. CSV and TSV, by contrast, do the following:
  * If there are too few keys, but these match the header, empty fields are emitted.
  * If there are too many keys, but these match the header up to the number of header fields, the extra fields are emitted.
  * If keys don't match the header, this is an error.

GENMD-RUN-COMMAND
cat data/under-over.json
GENMD-EOF

GENMD-RUN-COMMAND
mlr --ijson --ocsvlite cat data/under-over.json
GENMD-EOF

GENMD-RUN-COMMAND-TOLERATING-ERROR
mlr --ijson --ocsvlite cat data/key-change.json
GENMD-EOF

GENMD-RUN-COMMAND
mlr --ijson --ocsv cat data/under-over.json
GENMD-EOF

GENMD-RUN-COMMAND-TOLERATING-ERROR
mlr --ijson --ocsv cat data/key-change.json
GENMD-EOF

* In short, use-cases for CSV-lite and TSV-lite are often found when dealing with CSV/TSV files which are formatted in some non-standard way -- you have a little more flexibility available to you. (As an example of this flexibility: ASV and USV are nothing more than CSV-lite with different values for FS and RS.)

CSV, TSV, CSV-lite, and TSV-lite have in common the `--implicit-csv-header` flag for input and the `--headerless-csv-output` flag for output.

See also the [`--lazy-quotes` flag](reference-main-flag-list.md#csv-only-flags), which can help with CSV files that are not fully compliant with RFC-4180.

## JSON

[JSON](https://json.org) is a format which supports scalars (numbers, strings,
booleans, etc.) as well as "objects" (maps) and "arrays" (lists), while Miller
is a tool for handling **tabular data** only.  By *tabular JSON* I mean the
data is either a sequence of one or more objects, or an array consisting of one
or more objects.  Miller treats JSON objects as name-indexed records.

This means Miller cannot (and should not) handle arbitrary JSON.  In practice,
though, Miller can handle single JSON objects as well as lists of them. The only
kinds of JSON that are unmillerable are single scalars (e.g., file contents `3`)
and arrays of non-object (e.g., file contents `[1,2,3,4,5]`).  Check out
[jq](https://stedolan.github.io/jq/) for a tool that handles all valid JSON.

In short, if you have tabular data represented in JSON -- lists of objects,
either with or without outermost `[...]` -- [then Miller can handle that for
you.

### Single-level JSON objects

An **array of single-level objects** is, quite simply, **a table**:

GENMD-RUN-COMMAND
mlr --json head -n 2 then cut -f color,shape data/json-example-1.json
GENMD-EOF

GENMD-RUN-COMMAND
mlr --json head -n 2 then cut -f color,u,v data/json-example-1.json
GENMD-EOF

Single-level JSON data goes back and forth between JSON and tabular formats
in the direct way:

GENMD-RUN-COMMAND
mlr --ijson --opprint head -n 2 then cut -f color,u,v data/json-example-1.json
GENMD-EOF

GENMD-RUN-COMMAND
mlr --ijson --opprint cat data/json-example-1.json
GENMD-EOF

### Nested JSON objects

Additionally, Miller can **tabularize nested objects by concatenating keys**. If your processing has
input as well as output in JSON format, JSON structure is preserved throughout the processing:

GENMD-RUN-COMMAND
mlr --json head -n 2 data/json-example-2.json
GENMD-EOF

But if the input format is JSON and the output format is not (or vice versa), then key-concatenation applies:

GENMD-RUN-COMMAND
mlr --ijson --opprint head -n 4 data/json-example-2.json
GENMD-EOF

This is discussed in more detail on the page [Flatten/unflatten: JSON vs. tabular formats](flatten-unflatten.md).

Use `--jflatsep yourseparatorhere` to specify the string used for key concatenation: this defaults to a single dot.

### JSON-in-CSV

It's quite common to have CSV data that contains stringified JSON as a column.
See the [JSON parse and stringify section](reference-main-data-types.md#json-parse-and-stringify) for ways to
decode these in Miller.

## JSON Lines

[JSON Lines](https://jsonlines.org) is similar to JSON, except:

* UTF-8 encoding must be supported
* There is no outermost `[...]`
* Each record is on a single line

Miller handles this:

GENMD-RUN-COMMAND
mlr --icsv --ojson head -n 2 example.csv
GENMD-EOF

GENMD-RUN-COMMAND
mlr --icsv --ojsonl head -n 2 example.csv
GENMD-EOF

Note that for _input_ data, either is acceptable: whether you use `--ijson` or `--ijsonl`, Miller
won't reject your input data for lack of outermost `[...]`, nor will it reject your data for placement
of newlines. The difference is on _output_: using `--ojson`, you get outermost `[...]` and pretty-printed
records; using `--ojsonl`, you get no outermost `[...]`, and one line per record.

## PPRINT: Pretty-printed tabular

Miller's pretty-print format is similar to CSV, but with column alignment.  For example, compare

GENMD-RUN-COMMAND
mlr --ocsv cat data/small
GENMD-EOF

GENMD-RUN-COMMAND
mlr --opprint cat data/small
GENMD-EOF

Note that while Miller is a line-at-a-time processor and retains input lines in memory only where necessary (e.g., for sort), pretty-print output requires it to accumulate all input lines (so that it can compute maximum column widths) before producing any output. This has two consequences: (a) Pretty-print output will not work in `tail -f` contexts, where Miller will be waiting for an end-of-file marker that never arrives; (b) Pretty-print output for large files is constrained by the available machine memory.

See [Record Heterogeneity](record-heterogeneity.md) for how Miller handles changes of field names within a single data stream.

Since Miller 5.0.0, you can use `--barred` or `--barred-output` with pprint output format:

GENMD-RUN-COMMAND
mlr --opprint --barred cat data/small
GENMD-EOF

Since Miller 6.11.0, you can use `--barred-input` with pprint input format:

GENMD-RUN-COMMAND
mlr -o pprint --barred cat data/small | mlr -i pprint --barred-input -o json filter '$b == "pan"'
GENMD-EOF

## Markdown tabular

Markdown format looks like this:

GENMD-RUN-COMMAND
mlr --omd cat data/small
GENMD-EOF

which renders like this when dropped into various web tools (e.g. github.comments):

![pix/omd.png](pix/omd.png)

As of Miller 4.3.0, markdown format is supported only for output, not input; as of Miller 6.11.0, markdown format
is supported for input as well.

## XTAB: Vertical tabular

This is perhaps most useful for looking a very wide and/or multi-column data which causes line-wraps on the screen (but see also
[ngrid](https://github.com/twosigma/ngrid/) for an entirely different, very powerful option). Namely:

GENMD-CARDIFY-HIGHLIGHT-ONE
$ grep -v '^#' /etc/passwd | head -n 6 | mlr --nidx --fs : --opprint cat
1          2 3  4  5                          6               7
nobody     * -2 -2 Unprivileged User          /var/empty      /usr/bin/false
root       * 0  0  System Administrator       /var/root       /bin/sh
daemon     * 1  1  System Services            /var/root       /usr/bin/false
_uucp      * 4  4  Unix to Unix Copy Protocol /var/spool/uucp /usr/sbin/uucico
_taskgated * 13 13 Task Gate Daemon           /var/empty      /usr/bin/false
_networkd  * 24 24 Network Services           /var/networkd   /usr/bin/false
GENMD-EOF

GENMD-CARDIFY-HIGHLIGHT-ONE
$ grep -v '^#' /etc/passwd | head -n 2 | mlr --nidx --fs : --oxtab cat
1 nobody
2 *
3 -2
4 -2
5 Unprivileged User
6 /var/empty
7 /usr/bin/false

1 root
2 *
3 0
4 0
5 System Administrator
6 /var/root
7 /bin/sh
GENMD-EOF

GENMD-CARDIFY-HIGHLIGHT-THREE
$ grep -v '^#' /etc/passwd | head -n 2 | \
  mlr --nidx --fs : --ojson \
    label name,password,uid,gid,gecos,home_dir,shell
[
{
  "name": "nobody",
  "password": "*",
  "uid": -2,
  "gid": -2,
  "gecos": "Unprivileged User",
  "home_dir": "/var/empty",
  "shell": "/usr/bin/false"
},
{
  "name": "root",
  "password": "*",
  "uid": 0,
  "gid": 0,
  "gecos": "System Administrator",
  "home_dir": "/var/root",
  "shell": "/bin/sh"
}
]
GENMD-EOF

## DKVP: Key-value pairs

Miller's default file format is DKVP, for **delimited key-value pairs**. Example:

GENMD-RUN-COMMAND
mlr cat data/small
GENMD-EOF

Such data is easy to generate, e.g., in Ruby with

GENMD-CARDIFY
puts "host=#{hostname},seconds=#{t2-t1},message=#{msg}"
GENMD-EOF

GENMD-CARDIFY
puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',')
GENMD-EOF

or `print` statements in various languages, e.g.

GENMD-CARDIFY
echo "type=3,user=$USER,date=$date\n";
GENMD-EOF

GENMD-CARDIFY
logger.log("type=3,user=$USER,date=$date\n");
GENMD-EOF

Fields lacking an IPS will have positional index (starting at 1) used as the key, as in NIDX format. For example, `dish=7,egg=8,flint` is parsed as `"dish" => "7", "egg" => "8", "3" => "flint"` and `dish,egg,flint` is parsed as `"1" => "dish", "2" => "egg", "3" => "flint"`.

As discussed in [Record Heterogeneity](record-heterogeneity.md), Miller handles changes of field names within the same data stream. But using DKVP format, this is particularly natural. One of my favorite use-cases for Miller is in application/server logs, where I log all sorts of lines such as

GENMD-CARDIFY
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100, resource=/path/to/file
resource=/some/other/path,loadsec=0.97,ok=false
GENMD-EOF

etc., and I log them as needed. Then later, I can use `grep`, `mlr --opprint group-like`, etc. to analyze my logs.

See the [separators page](reference-main-separators.md) regarding how to specify separators other than the default equals sign and comma.

## NIDX: Index-numbered (toolkit style)

With `--inidx --ifs ' ' --repifs`, Miller splits lines on spaces and assigns integer field names starting with 1.

This recapitulates Unix-toolkit behavior.

Example with index-numbered output:

GENMD-RUN-COMMAND
cat data/small
GENMD-EOF

GENMD-RUN-COMMAND
mlr --onidx --ofs ' ' cat data/small
GENMD-EOF

Example with index-numbered input:

GENMD-RUN-COMMAND
cat data/mydata.txt
GENMD-EOF

GENMD-RUN-COMMAND
mlr --inidx --ifs ' ' --odkvp cat data/mydata.txt
GENMD-EOF

Example with index-numbered input and output:

GENMD-RUN-COMMAND
cat data/mydata.txt
GENMD-EOF

GENMD-RUN-COMMAND
mlr --nidx --fs ' ' --repifs cut -f 2,3 data/mydata.txt
GENMD-EOF

## Data-conversion keystroke-savers

While you can do format conversion using `mlr --icsv --ojson cat myfile.csv`, there are also keystroke-savers for this purpose, such as `mlr --c2j cat myfile.csv`.  For a complete list:

GENMD-RUN-COMMAND
mlr help format-conversion-keystroke-saver-flags
GENMD-EOF

## Comments in data

You can include comments within your data files, and either have them ignored or passed directly through to the standard output as soon as they are encountered:

GENMD-RUN-COMMAND
mlr help comments-in-data-flags
GENMD-EOF

Examples:

GENMD-RUN-COMMAND
cat data/budget.csv
GENMD-EOF

GENMD-RUN-COMMAND
mlr --skip-comments --icsv --opprint sort -nr quantity data/budget.csv
GENMD-EOF

GENMD-RUN-COMMAND
mlr --pass-comments --icsv --opprint sort -nr quantity data/budget.csv
GENMD-EOF
