# CSV, with and without headers

## Headerless CSV on input or output

Sometimes we get CSV files which lack a header. For example, [data/headerless.csv](./data/headerless.csv):

GENMD-RUN-COMMAND
cat data/headerless.csv
GENMD-EOF

You can use Miller to add a header. The `--implicit-csv-header` applies positionally indexed labels:

GENMD-RUN-COMMAND
mlr --csv --implicit-csv-header cat data/headerless.csv
GENMD-EOF

Following that, you can rename the positionally indexed labels to names with meaning for your context.  For example:

GENMD-RUN-COMMAND
mlr --csv --implicit-csv-header label name,age,status data/headerless.csv
GENMD-EOF

Likewise, if you need to produce CSV which is lacking its header, you can pipe Miller's output to the system command `sed 1d`, or you can use Miller's `--headerless-csv-output` option:

GENMD-RUN-COMMAND
head -5 data/colored-shapes.dkvp | mlr --ocsv cat
GENMD-EOF

GENMD-RUN-COMMAND
head -5 data/colored-shapes.dkvp | mlr --ocsv --headerless-csv-output cat
GENMD-EOF

Lastly, often we say "CSV" or "TSV" when we have positionally indexed data in columns which are separated by commas or tabs, respectively. In this case it's perhaps simpler to **just use NIDX format** which was designed for this purpose. (See also [File Formats](file-formats.md).) For example:

GENMD-RUN-COMMAND
mlr --inidx --ifs comma --oxtab cut -f 1,3 data/headerless.csv
GENMD-EOF

## Headerless CSV with duplicate field values

Miller's default treatment of CSV data, in the absence of any other adjectives, is always _CSV with header_.
However, lots of folks think of CSV data -- comma-separated values -- as just that, values, with no header.

Here's some sample CSV data which is values-only, i.e. headerless:

GENMD-RUN-COMMAND
cat data/nas.csv
GENMD-EOF

There are clearly nine fields here, but if we try to have Miller parse it as CSV, we
see something happened:

GENMD-RUN-COMMAND
mlr --csv cat data/nas.csv
GENMD-EOF

What happened?

Miller is (by central design) a mapping from name to value, rather than integer
position to value as in most tools in the Unix toolkit such as `sort`, `cut`,
`awk`, etc. And its default behavior with repeated column/field names is to
append `_2`, `_3`, etc to dedupe them.  So given input `Yea=1,Yea=2` on the
same input line, first `Yea=1` is stored, then `Yea_2=2`. This is in the
input-parser.

Here, the first data line is being seen as a header line, and the repeated `NA`
values are being seen as duplicate keys that need to be deduplicated.

One solution is to use `--implicit-csv-header`, or its shorter alias `--hi`:

GENMD-RUN-COMMAND
mlr --csv --hi cat data/nas.csv
GENMD-EOF

Another solution is to use [NIDX format](file-formats.md#nidx-index-numbered-toolkit-style):

GENMD-RUN-COMMAND
mlr --inidx --ifs comma --ocsv cat data/nas.csv
GENMD-EOF

Either way, since there is no explicit header, fields are named `1` through `9`. We can use the
[label verb](reference-verbs.md#label) to apply more meaningful namees:

GENMD-RUN-COMMAND
mlr --csv --hi cat then label xsn,ysn,x,y,t,a,e29,e31,e32 data/nas.csv
GENMD-EOF

GENMD-RUN-COMMAND
mlr --inidx --ifs comma --ocsv cat then label xsn,ysn,x,y,t,a,e29,e31,e32 data/nas.csv
GENMD-EOF

## Regularizing ragged CSV

Miller handles [RFC-4180-compliant CSV](file-formats.md#csvtsvasvusvetc): in particular, it's an error if the number of data fields in a given data line don't match the number of header lines. But in the event that you have a CSV file in which some lines have less than the full number of fields, you can use Miller to pad them out. The trick is to use NIDX format, for which each line stands on its own without respect to a header line.

GENMD-RUN-COMMAND
cat data/ragged.csv
GENMD-EOF

GENMD-RUN-COMMAND
mlr --from data/ragged.csv --fs comma --nidx put '
  @maxnf = max(@maxnf, NF);
  @nf = NF;
  while(@nf < @maxnf) {
    @nf += 1;
    $[@nf] = ""
  }
'
GENMD-EOF

or, more simply,

GENMD-RUN-COMMAND
mlr --from data/ragged.csv --fs comma --nidx put '
  @maxnf = max(@maxnf, NF);
  while(NF < @maxnf) {
    $[NF+1] = "";
  }
'
GENMD-EOF

See also the [record-heterogeneity page](record-heterogeneity.md).
