# Record-heterogeneity
Miller documentation
Keywords: John Kerl, Kerl, Miller, miller, mlr, OLAP, data analysis software, regression, correlation, variance, data tools,

<titleinbody>Miller</titleinbody>
• About Miller
• File formats
• Miller features in the context of the Unix toolkit
• Record-heterogeneity
• Reference
• Data examples
• Cookbook
• FAQ
• Internationalization
• Compiling, portability, dependencies, and testing
• Performance
• Why C?
• Why call it Miller?
• How original is Miller?
• Things to do
• Documents by release
• Contact information
• GitHub repo

<titleinbody> Record-heterogeneity </titleinbody>

Contents:
• For I/O
• CSV and pretty-print
• Key-value-pair, vertical-tabular, and index-numbered formats
• For processing

We think of CSV tables as rectangular: if there are 17 columns in the header then there are 17 columns for every row, else the data have a formatting error.

But heterogeneous data abound (today’s no-SQL databases for example). Miller handles this.

# For I/O

## CSV and pretty-print

Miller simply prints a newline and a new header when there is a schema change. When there is no schema change, you get CSV per se as a special case. Likewise, Miller reads heterogeneous CSV or pretty-print input the same way. The difference between CSV and CSV-lite is that the former is RFC4180-compliant, while the latter readily handles heterogeneous data (which is non-compliant). For example:

    $ cat data/het.dkvp
    resource=/path/to/file,loadsec=0.45,ok=true
    record_count=100,resource=/path/to/file
    resource=/path/to/second/file,loadsec=0.32,ok=true
    record_count=150,resource=/path/to/second/file
    resource=/some/other/path,loadsec=0.97,ok=false

    $ mlr --ocsvlite cat data/het.dkvp
    resource,loadsec,ok
    /path/to/file,0.45,true

    record_count,resource
    100,/path/to/file

    resource,loadsec,ok
    /path/to/second/file,0.32,true

    record_count,resource
    150,/path/to/second/file

    resource,loadsec,ok
    /some/other/path,0.97,false

    $ mlr --opprint cat data/het.dkvp
    resource      loadsec ok
    /path/to/file 0.45    true

    record_count resource
    100          /path/to/file

    resource             loadsec ok
    /path/to/second/file 0.32    true

    record_count resource
    150          /path/to/second/file

    resource         loadsec ok
    /some/other/path 0.97    false

You may also find Miller’s `group-like` feature handy (see also Reference):

    $ mlr --ocsvlite group-like data/het.dkvp
    resource,loadsec,ok
    /path/to/file,0.45,true
    /path/to/second/file,0.32,true
    /some/other/path,0.97,false

    record_count,resource
    100,/path/to/file
    150,/path/to/second/file

    $ mlr --opprint group-like data/het.dkvp
    resource             loadsec ok
    /path/to/file        0.45    true
    /path/to/second/file 0.32    true
    /some/other/path     0.97    false

    record_count resource
    100          /path/to/file
    150          /path/to/second/file

## Key-value-pair, vertical-tabular, and index-numbered formats

For these formats, record-heterogeneity comes naturally:

    $ cat data/het.dkvp
    resource=/path/to/file,loadsec=0.45,ok=true
    record_count=100,resource=/path/to/file
    resource=/path/to/second/file,loadsec=0.32,ok=true
    record_count=150,resource=/path/to/second/file
    resource=/some/other/path,loadsec=0.97,ok=false

    $ mlr --onidx --ofs ' ' cat data/het.dkvp
    /path/to/file 0.45 true
    100 /path/to/file
    /path/to/second/file 0.32 true
    150 /path/to/second/file
    /some/other/path 0.97 false

    $ mlr --oxtab cat data/het.dkvp
    resource /path/to/file
    loadsec  0.45
    ok       true

    record_count 100
    resource     /path/to/file

    resource /path/to/second/file
    loadsec  0.32
    ok       true

    record_count 150
    resource     /path/to/second/file

    resource /some/other/path
    loadsec  0.97
    ok       false

    $ mlr --oxtab group-like data/het.dkvp
    resource /path/to/file
    loadsec  0.45
    ok       true

    resource /path/to/second/file
    loadsec  0.32
    ok       true

    resource /some/other/path
    loadsec  0.97
    ok       false

    record_count 100
    resource     /path/to/file

    record_count 150
    resource     /path/to/second/file

# For processing

Miller operates on specified fields and takes the rest along: for example, if you are sorting on the `count` field then all records in the input stream must have a `count` field but the other fields can vary, and moreover the sorted-on field name(s) don’t need to be in the same position on each line:

    $ cat data/sort-het.dkvp
    count=500,color=green
    count=600
    status=ok,count=250,hours=0.22
    status=ok,count=200,hours=3.4
    count=300,color=blue
    count=100,color=green
    count=450

    $ mlr sort -n count data/sort-het.dkvp
    count=100,color=green
    status=ok,count=200,hours=3.4
    status=ok,count=250,hours=0.22
    count=300,color=blue
    count=450
    count=500,color=green
    count=600