Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-26 09:23:55 +00:00
Code Issues Projects Releases Wiki Activity

Add FAQ entry 'Headerless CSV with duplicate field values'

Browse source 
See https://github.com/johnkerl/miller/issues/524#issuecomment-849571560
This commit is contained in:
Nikos Alexandris 2021-06-01 15:54:07 +03:00
parent 1d995de632
commit 7cb2d75b5c
2 changed files with 26 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

13
docs/faq.rst.in
View file

@ -15,6 +15,19 @@ Check the field-separators of the data, e.g. with the command-line ``head`` prog
Also try ``od -xcv`` and/or ``cat -e`` on your file to check for non-printable characters.
Headerless CSV with duplicate field values
------------------------------------------
Miller is (by central design) a mapping from name to value, rather than integer position to value as in most tools in the Unix toolkit such as sort, cut, awk, etc. So given input ``Yea=1,Yea=2`` on the same input line, first ``Yea=1`` is stored, then updated with ``Yea=2``. This is in the input-parser and the value ``Yea=1`` is unavailable to any further processing. The following example line comes from a headerless CSV file and includes 5 times the string (value) ``'NA'``:
``ag '0.9' nas.csv |head -1
2:-349801.10097848,4537221.43295653,2,1,NA,NA,NA,NA,NA``
The repeated ``'NA'`` strings (values) in the same line will be treated by miller as fields (or columns for the matter) with same name, thus only one is kept in the output.
This can be worked-around by telling ``mlr`` that there is no header row by using ``--implicit-csv-header`` or changing the input format by using ``nidx`` like so:
``ag '0.9' nas.csv | mlr --n2c --fs "," label xsn,ysn,x,y,t,a,e29,e31,e32 then head``
Diagnosing delimiter specifications
----------------------------------------------------------------

13
docs6/faq.rst.in
View file

@ -15,6 +15,19 @@ Check the field-separators of the data, e.g. with the command-line ``head`` prog
Also try ``od -xcv`` and/or ``cat -e`` on your file to check for non-printable characters.
Headerless CSV with duplicate field values
------------------------------------------
Miller is (by central design) a mapping from name to value, rather than integer position to value as in most tools in the Unix toolkit such as sort, cut, awk, etc. So given input ``Yea=1,Yea=2`` on the same input line, first ``Yea=1`` is stored, then updated with ``Yea=2``. This is in the input-parser and the value ``Yea=1`` is unavailable to any further processing. The following example line comes from a headerless CSV file and includes 5 times the string (value) ``'NA'``:
``ag '0.9' nas.csv |head -1
2:-349801.10097848,4537221.43295653,2,1,NA,NA,NA,NA,NA``
The repeated ``'NA'`` strings (values) in the same line will be treated by miller as fields (or columns for the matter) with same name, thus only one is kept in the output.
This can be worked-around by telling ``mlr`` that there is no header row by using ``--implicit-csv-header`` or changing the input format by using ``nidx`` like so:
``ag '0.9' nas.csv | mlr --n2c --fs "," label xsn,ysn,x,y,t,a,e29,e31,e32 then head``
Diagnosing delimiter specifications
----------------------------------------------------------------