# Shapes of data

## No output at all

Try `od -xcv` and/or `cat -e` on your file to check for non-printable characters.

If you're using Miller version less than 5.0.0 (try `mlr --version` on your system to find out), when the line-ending-autodetect feature was introduced, please see [http://johnkerl.org/miller-releases/miller-4.5.0/doc/index.html](http://johnkerl.org/miller-releases/miller-4.5.0/doc/index.html).

## Fields not selected

Check the field-separators of the data, e.g. with the command-line `head` program. Example: for CSV, Miller's default record separator is comma; if your data is tab-delimited, e.g. `aTABbTABc`, then Miller won't find three fields named `a`, `b`, and `c` but rather just one named `aTABbTABc`.  Solution in this case: `mlr --fs tab {remaining arguments ...}`.

Also try `od -xcv` and/or `cat -e` on your file to check for non-printable characters.

## Diagnosing delimiter specifications

Use the `file` command to see if there are CR/LF terminators (in this case, there are not):

GENMD-CARDIFY-HIGHLIGHT-ONE
file data/colours.csv
data/colours.csv: Unicode text, UTF-8 text
GENMD-EOF

Look at the file to find names of fields:

GENMD-CARDIFY-HIGHLIGHT-ONE
cat data/colours.csv
KEY;DE;EN;ES;FI;FR;IT;NL;PL;TO;TR
masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz
masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah
GENMD-EOF

Extract a few fields:

GENMD-CARDIFY-HIGHLIGHT-ONE
mlr --csv cut -f KEY,PL,TO data/colours.csv
GENMD-EOF

Use XTAB output format to get a sharper picture of where records/fields are being split:

GENMD-CARDIFY-HIGHLIGHT-ONE
mlr --icsv --oxtab cat data/colours.csv
KEY;DE;EN;ES;FI;FR;IT;NL;PL;TO;TR masterdata_colourcode_1;Weiß;White;Blanco;Valkoinen;Blanc;Bianco;Wit;Biały;Alb;Beyaz

KEY;DE;EN;ES;FI;FR;IT;NL;PL;TO;TR masterdata_colourcode_2;Schwarz;Black;Negro;Musta;Noir;Nero;Zwart;Czarny;Negru;Siyah
GENMD-EOF

Using XTAB output format makes it clearer that `KEY;DE;...;TR` is being treated as a single field name in the CSV header, and likewise each subsequent line is being treated as a single field value. This is because the default field separator is a comma but we have semicolons here.  Use XTAB again with different field separator (`--fs semicolon`):

GENMD-CARDIFY-HIGHLIGHT-ONE
mlr --icsv --ifs semicolon --oxtab cat data/colours.csv
KEY masterdata_colourcode_1
DE  Weiß
EN  White
ES  Blanco
FI  Valkoinen
FR  Blanc
IT  Bianco
NL  Wit
PL  Biały
TO  Alb
TR  Beyaz

KEY masterdata_colourcode_2
DE  Schwarz
EN  Black
ES  Negro
FI  Musta
FR  Noir
IT  Nero
NL  Zwart
PL  Czarny
TO  Negru
TR  Siyah
GENMD-EOF

Using the new field-separator, retry the cut:

GENMD-CARDIFY-HIGHLIGHT-ONE
mlr --csv --fs semicolon cut -f KEY,PL,TO data/colours.csv
KEY;PL;TO
masterdata_colourcode_1;Biały;Alb
masterdata_colourcode_2;Czarny;Negru
GENMD-EOF

## I assigned $9 and it's not 9th

Miller records are ordered lists of key-value pairs. For NIDX format, DKVP format when keys are missing, or CSV/CSV-lite format with `--implicit-csv-header`, Miller will sequentially assign keys of the form `1`, `2`, etc. But these are not integer array indices: they're just field names taken from the initial field ordering in the input data, when it was originally read from the input file(s).

GENMD-RUN-COMMAND
echo x,y,z | mlr --dkvp cat
GENMD-EOF

GENMD-RUN-COMMAND
echo x,y,z | mlr --dkvp put '$6="a";$4="b";$55="cde"'
GENMD-EOF

GENMD-RUN-COMMAND
echo x,y,z | mlr --nidx cat
GENMD-EOF

GENMD-RUN-COMMAND
echo x,y,z | mlr --csv --implicit-csv-header cat
GENMD-EOF

GENMD-RUN-COMMAND
echo x,y,z | mlr --dkvp rename 2,999
GENMD-EOF

GENMD-RUN-COMMAND
echo x,y,z | mlr --dkvp rename 2,newname
GENMD-EOF

GENMD-RUN-COMMAND
echo x,y,z | mlr --csv --implicit-csv-header reorder -f 3,1,2
GENMD-EOF

## Why doesn't mlr cut put fields in the order I want?

Example: columns `rate,shape,flag` were requested but they appear here in the order `shape,flag,rate`:

GENMD-RUN-COMMAND
cat example.csv
GENMD-EOF

GENMD-RUN-COMMAND
mlr --csv cut -f rate,shape,flag example.csv
GENMD-EOF

The issue is that Miller's `cut`, by default, outputs cut fields in the order they appear in the input data. This design decision was made intentionally to parallel the Unix/Linux system `cut` command, which has the same semantics.

The solution is to use the `-o` option:

GENMD-RUN-COMMAND
mlr --csv cut -o -f rate,shape,flag example.csv
GENMD-EOF

## Numbering and renumbering records

The `awk`-like built-in variable `NR` is incremented for each input record:

GENMD-RUN-COMMAND
cat example.csv
GENMD-EOF

GENMD-RUN-COMMAND
mlr --csv put '$nr = NR' example.csv
GENMD-EOF

However, this is the record number within the original input stream -- not after any filtering you may have done:

GENMD-RUN-COMMAND
mlr --csv filter '$color == "yellow"' then put '$nr = NR' example.csv
GENMD-EOF

There are two good options here. One is to use the `cat` verb with `-n`:

GENMD-RUN-COMMAND
mlr --csv filter '$color == "yellow"' then cat -n example.csv
GENMD-EOF

The other is to keep your own counter within the `put` DSL:

GENMD-RUN-COMMAND
mlr --csv filter '$color == "yellow"' then put 'begin {@n = 1} $n = @n; @n += 1' example.csv
GENMD-EOF

The difference is a matter of taste (although `mlr cat -n` puts the counter first).

## Options for dealing with duplicate rows

If your data has records appearing multiple times, you can use [uniq](reference-verbs.md#uniq) to show and/or count the unique records.

If you want to look at partial uniqueness -- for example, show only the first record for each unique combination of the `account_id` and `account_status` fields -- you might use `mlr head -n 1 -g account_id,account_status`. Please also see [head](reference-verbs.md#head).

## Rectangularizing data

Suppose you have a method (in whatever language) which is printing things of the form

GENMD-INCLUDE-ESCAPED(data/rect-outer.txt)

and then calls another method which prints things of the form

GENMD-INCLUDE-ESCAPED(data/rect-middle.txt)

and then, perhaps, that second method calls a third method which prints things of the form

GENMD-INCLUDE-ESCAPED(data/rect-inner.txt)

with the result that your program's output is

GENMD-INCLUDE-ESCAPED(data/rect.txt)

The idea here is that middles starting with a 1 belong to the outer value of 1, and so on.  (For example, the outer values might be account IDs, the middle values might be invoice IDs, and the inner values might be invoice line-items.) If you want all the middle and inner lines to have the context of which outers they belong to, you can modify your software to pass all those through your methods. Alternatively, don't refactor your code just to handle some ad-hoc log-data formatting -- instead, use the following to [rectangularize the data](record-heterogeneity.md). The idea is to use an out-of-stream variable to accumulate fields across records. Clear that variable when you see an outer ID; accumulate fields; emit output when you see the inner IDs.

GENMD-RUN-COMMAND
mlr --from data/rect.txt put -q '
  is_present($outer) {
    unset @r
  }
  for (k, v in $*) {
    @r[k] = v
  }
  is_present($inner1) {
    emit @r
  }'
GENMD-EOF

See also the [record-heterogeneity page](record-heterogeneity.md); see in
particular the [`regularize` verb](reference-verbs.md#regularize) for a way to
do this with much less keystroking.
