From 7cb2d75b5c3202ca60586e8fb460f0fa3fe14249 Mon Sep 17 00:00:00 2001 From: Nikos Alexandris Date: Tue, 1 Jun 2021 15:54:07 +0300 Subject: [PATCH] Add FAQ entry 'Headerless CSV with duplicate field values' See https://github.com/johnkerl/miller/issues/524#issuecomment-849571560 --- docs/faq.rst.in | 13 +++++++++++++ docs6/faq.rst.in | 13 +++++++++++++ 2 files changed, 26 insertions(+) diff --git a/docs/faq.rst.in b/docs/faq.rst.in index 10238fe19..18c14e1d1 100644 --- a/docs/faq.rst.in +++ b/docs/faq.rst.in @@ -15,6 +15,19 @@ Check the field-separators of the data, e.g. with the command-line ``head`` prog Also try ``od -xcv`` and/or ``cat -e`` on your file to check for non-printable characters. +Headerless CSV with duplicate field values +------------------------------------------ + +Miller is (by central design) a mapping from name to value, rather than integer position to value as in most tools in the Unix toolkit such as sort, cut, awk, etc. So given input ``Yea=1,Yea=2`` on the same input line, first ``Yea=1`` is stored, then updated with ``Yea=2``. This is in the input-parser and the value ``Yea=1`` is unavailable to any further processing. The following example line comes from a headerless CSV file and includes 5 times the string (value) ``'NA'``: + +``ag '0.9' nas.csv |head -1 +2:-349801.10097848,4537221.43295653,2,1,NA,NA,NA,NA,NA`` + +The repeated ``'NA'`` strings (values) in the same line will be treated by miller as fields (or columns for the matter) with same name, thus only one is kept in the output. + +This can be worked-around by telling ``mlr`` that there is no header row by using ``--implicit-csv-header`` or changing the input format by using ``nidx`` like so: +``ag '0.9' nas.csv | mlr --n2c --fs "," label xsn,ysn,x,y,t,a,e29,e31,e32 then head`` + Diagnosing delimiter specifications ---------------------------------------------------------------- diff --git a/docs6/faq.rst.in b/docs6/faq.rst.in index d1231f885..b68b7b143 100644 --- a/docs6/faq.rst.in +++ b/docs6/faq.rst.in @@ -15,6 +15,19 @@ Check the field-separators of the data, e.g. with the command-line ``head`` prog Also try ``od -xcv`` and/or ``cat -e`` on your file to check for non-printable characters. +Headerless CSV with duplicate field values +------------------------------------------ + +Miller is (by central design) a mapping from name to value, rather than integer position to value as in most tools in the Unix toolkit such as sort, cut, awk, etc. So given input ``Yea=1,Yea=2`` on the same input line, first ``Yea=1`` is stored, then updated with ``Yea=2``. This is in the input-parser and the value ``Yea=1`` is unavailable to any further processing. The following example line comes from a headerless CSV file and includes 5 times the string (value) ``'NA'``: + +``ag '0.9' nas.csv |head -1 +2:-349801.10097848,4537221.43295653,2,1,NA,NA,NA,NA,NA`` + +The repeated ``'NA'`` strings (values) in the same line will be treated by miller as fields (or columns for the matter) with same name, thus only one is kept in the output. + +This can be worked-around by telling ``mlr`` that there is no header row by using ``--implicit-csv-header`` or changing the input format by using ``nidx`` like so: +``ag '0.9' nas.csv | mlr --n2c --fs "," label xsn,ysn,x,y,t,a,e29,e31,e32 then head`` + Diagnosing delimiter specifications ----------------------------------------------------------------