mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-19 09:24:14 +00:00
75 lines
3.4 KiB
HTML
75 lines
3.4 KiB
HTML
POKI_PUT_TOC_HERE
|
|
|
|
<h1>No output at all</h1>
|
|
|
|
<p/>Check the line-terminators of the data, e.g. with the command-line
|
|
<tt>file</tt> program. Example: for CSV, Miller’s default line terminator
|
|
is CR/LF (carriage return followed by linefeed, following
|
|
<a href="https://tools.ietf.org/html/rfc4180">RFC4180</a>). Yet if your CSV has
|
|
*nix-standard LF line endings, Miller will keep reading the file looking for a
|
|
CR/LF which never appears. Solution in this case: tell Miller the input has LF line-terminator, e.g. <b>mlr --csv --rs
|
|
lf {remaining arguments ...}</b>.
|
|
|
|
<p/>Also try <tt>od -xcv</tt> and/or <tt>cat -e</tt> on your file to check for non-printable characters.
|
|
|
|
<h1>Fields not selected</h1>
|
|
|
|
<p/>Check the field-separators of the data, e.g. with the command-line
|
|
<tt>head</tt> program. Example: for CSV, Miller’s default record
|
|
separator is comma; if your data is tab-delimited, e.g. <tt>aTABbTABc</tt>,
|
|
then Miller won’t find three fields named <tt>a</tt>, <tt>b</tt>, and
|
|
<tt>c</tt> but rather just one named <tt>aTABbTABc</tt>. Solution in this
|
|
case: <tt>mlr --fs tab {remaining arguments ...}</tt>.
|
|
|
|
<p/>Also try <tt>od -xcv</tt> and/or <tt>cat -e</tt> on your file to check for non-printable characters.
|
|
|
|
<h1>Diagnosing delimiter specifications</h1>
|
|
|
|
POKI_INCLUDE_ESCAPED(data/delimiter-examples.txt)HERE
|
|
|
|
<h1>Error-output in certain string cases</h1>
|
|
|
|
<p/> <tt>mlr put '$y = string($x); $z=$y.$y'</tt> gives <tt>(error)</tt> on
|
|
numeric data such as <tt>x=123</tt> while <tt>mlr put
|
|
'$z=string($x).string($x)'</tt> does not. This is because in the former case
|
|
<tt>y</tt> is computed and stored as a string, then re-parsed as an integer,
|
|
for which string-concatenation is an invalid operator.
|
|
|
|
<h1>How do I parse log-file output?</h1>
|
|
|
|
<p/>Suppose you have log-file lines such as
|
|
|
|
POKI_CARDIFY(2015-10-08 08:29:09,445 INFO com.company.path.to.ClassName @ [sometext] various/sorts/of data {& punctuation} hits=1 status=0 time=2.378)HERE
|
|
|
|
I prefer to pre-filter with <tt>grep</tt> and/or <tt>sed</tt> to extract the structured text, then hand that to Miller. Example:
|
|
|
|
POKI_CARDIFY(grep 'various sorts' *.log | sed 's/.*} //' | mlr --fs space --repifs --oxtab stats1 -a min,p10,p50,p90,max -f time -g status)HERE
|
|
|
|
<h1>How do I examine then-chaining?</h1>
|
|
|
|
<p/>Then-chaining found in Miller is intended to function the same as Unix
|
|
pipes. You can print your data one pipeline step at a time, to see what
|
|
intermediate
|
|
output at one step becomes the input to the next step.
|
|
|
|
<p/>First, review the input data:
|
|
|
|
POKI_RUN_COMMAND{{cat data/then-example.csv}}HERE
|
|
|
|
Next, run the first step of your command, omitting anything from the first <tt>then</tt> onward:
|
|
|
|
POKI_RUN_COMMAND{{mlr --icsv --rs lf --opprint count-distinct -f Status,Payment_Type data/then-example.csv}}HERE
|
|
|
|
After that, run it with the next <tt>then</tt> step included:
|
|
|
|
POKI_RUN_COMMAND{{mlr --icsv --rs lf --opprint count-distinct -f Status,Payment_Type then sort -nr count data/then-example.csv}}HERE
|
|
|
|
Now if you include another <tt>then</tt> step after this, the columns <tt>Status</tt>,
|
|
<tt>Payment_Type</tt>, and <tt>count</tt> will be its input.
|
|
|
|
<p/>Note, by the way, that you’ll get the same results using pipes:
|
|
POKI_RUN_COMMAND{{mlr --csv --rs lf count-distinct -f Status,Payment_Type data/then-example.csv | mlr --icsv --rs lf --opprint sort -nr count}}HERE
|
|
|
|
<h1>How do I do arithmetic on fields with currency symbols?</h1>
|
|
|
|
POKI_INCLUDE_ESCAPED(data/dollar-sign.txt)HERE
|