Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-19 09:24:14 +00:00
Code Issues Projects Releases Wiki Activity
miller/docs/reference-verbs.rst.in
John Kerl 897ea53168 lighten code-card background
2020-09-30 21:46:43 -04:00

1270 lines
30 KiB
ReStructuredText
Raw Blame History

Verbs reference
================================================================
Overview
----------------------------------------------------------------
When you type ``mlr {something} myfile.dat``, the ``{something}`` part is called a **verb**. It specifies how you want to transform your data. (See also :ref:`reference-command-overview` for a breakdown.) The following is an alphabetical list of verbs with their descriptions.
The verbs ``put`` and ``filter`` are special in that they have a rich expression language (domain-specific language, or "DSL"). More information about them can be found at :doc:`reference-dsl`.
Here's a comparison of verbs and ``put``/``filter`` DSL expressions:
Example:
::
POKI_RUN_COMMAND{{mlr stats1 -a sum -f x -g a data/small}}HERE
* Verbs are coded in C
* They run a bit faster
* They take fewer keystrokes
* There is less to learn
* Their customization is limited to each verb's options
Example:
::
POKI_RUN_COMMAND{{mlr put -q '@x_sum[$a] += $x; end{emit @x_sum, "a"}' data/small}}HERE
* You get to write your own DSL expressions
* They run a bit slower
* They take more keystrokes
* There is more to learn
* They are highly customizable
.. _reference-verbs-altkv:
altkv
----------------------------------------------------------------
Map list of values to alternating key/value pairs.
::
POKI_RUN_COMMAND{{mlr altkv -h}}HERE
::
POKI_RUN_COMMAND{{echo 'a,b,c,d,e,f' | mlr altkv}}HERE
::
POKI_RUN_COMMAND{{echo 'a,b,c,d,e,f,g' | mlr altkv}}HERE
.. _reference-verbs-bar:
bar
----------------------------------------------------------------
Cheesy bar-charting.
::
POKI_RUN_COMMAND{{mlr bar -h}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint bar --lo 0 --hi 1 -f x,y data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint bar --lo 0.4 --hi 0.6 -f x,y data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint bar --auto -f x,y data/small}}HERE
.. _reference-verbs-bootstrap:
bootstrap
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr bootstrap --help}}HERE
The canonical use for bootstrap sampling is to put error bars on statistical quantities, such as mean. For example:
::
$ mlr --opprint stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color u_mean u_count
yellow 0.497129 1413
red 0.492560 4641
purple 0.494005 1142
green 0.504861 1109
blue 0.517717 1470
orange 0.490532 303
::
$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color u_mean u_count
yellow 0.500651 1380
purple 0.501556 1111
green 0.503272 1068
red 0.493895 4702
blue 0.512529 1496
orange 0.521030 321
::
$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color u_mean u_count
yellow 0.498046 1485
blue 0.513576 1417
red 0.492870 4595
orange 0.507697 307
green 0.496803 1075
purple 0.486337 1199
::
$ mlr --opprint bootstrap then stats1 -a mean,count -f u -g color data/colored-shapes.dkvp
color u_mean u_count
blue 0.522921 1447
red 0.490717 4617
yellow 0.496450 1419
purple 0.496523 1192
green 0.507569 1111
orange 0.468014 292
.. _reference-verbs-cat:
cat
----------------------------------------------------------------
Most useful for format conversions (see :doc:`file-formats`, and concatenating multiple same-schema CSV files to have the same header:
::
POKI_RUN_COMMAND{{mlr cat -h}}HERE
::
POKI_RUN_COMMAND{{cat data/a.csv}}HERE
::
POKI_RUN_COMMAND{{cat data/b.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --csv cat data/a.csv data/b.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --icsv --oxtab cat data/a.csv data/b.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --csv cat -n data/a.csv data/b.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint cat -n -g a data/small}}HERE
.. _reference-verbs-check:
check
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr check --help}}HERE
.. _reference-verbs-clean-whitespace:
clean-whitespace
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr clean-whitespace --help}}HERE
::
POKI_RUN_COMMAND{{mlr --icsv --ojson cat data/clean-whitespace.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --icsv --ojson clean-whitespace -k data/clean-whitespace.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --icsv --ojson clean-whitespace -v data/clean-whitespace.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --icsv --ojson clean-whitespace data/clean-whitespace.csv}}HERE
Function links:
* :ref:`reference-dsl-lstrip`
* :ref:`reference-dsl-rstrip`
* :ref:`reference-dsl-strip`
* :ref:`reference-dsl-collapse_whitespace`
* :ref:`reference-dsl-clean_whitespace`
.. _reference-verbs-count:
count
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr count --help}}HERE
::
POKI_RUN_COMMAND{{mlr count data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr count -g a data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr count -n -g a data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr count -g b data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr count -n -g b data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr count -g a,b data/medium}}HERE
.. _reference-verbs-count-distinct:
count-distinct
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr count-distinct --help}}HERE
::
POKI_RUN_COMMAND{{mlr count-distinct -f a,b then sort -nr count data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr count-distinct -u -f a,b data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr count-distinct -f a,b -o someothername then sort -nr someothername data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr count-distinct -n -f a,b data/medium}}HERE
.. _reference-verbs-count-similar:
count-similar
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr count-similar --help}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint head -n 20 data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint head -n 20 then count-similar -g a data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint head -n 20 then count-similar -g a then sort -f a data/medium}}HERE
.. _reference-verbs-cut:
cut
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr cut --help}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint cut -f y,x,i data/small}}HERE
::
POKI_RUN_COMMAND{{echo 'a=1,b=2,c=3' | mlr cut -f b,c,a}}HERE
::
POKI_RUN_COMMAND{{echo 'a=1,b=2,c=3' | mlr cut -o -f b,c,a}}HERE
.. _reference-verbs-decimate:
decimate
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr decimate --help}}HERE
.. _reference-verbs-fill-down:
fill-down
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr fill-down --help}}HERE
::
POKI_RUN_COMMAND{{cat data/fill-down.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --csv fill-down -f b data/fill-down.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --csv fill-down -a -f b data/fill-down.csv}}HERE
.. _reference-verbs-filter:
filter
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr filter --help}}HERE
Features which filter shares with put
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please see :doc:`reference-dsl` for more information about the expression language for ``mlr filter``.
.. _reference-verbs-format-values:
format-values
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr format-values --help}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint format-values data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint format-values -n data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint format-values -i %08llx -f %.6le -s X%sX data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint format-values -i %08llx -f %.6le -s X%sX -n data/small}}HERE
.. _reference-verbs-fraction:
fraction
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr fraction --help}}HERE
For example, suppose you have the following CSV file:
::
POKI_INCLUDE_ESCAPED(data/fraction-example.csv)HERE
Then we can see what each record's ``n`` contributes to the total ``n``:
::
POKI_RUN_COMMAND{{mlr --opprint fraction -f n data/fraction-example.csv}}HERE
Using ``-g`` we can split those out by gender, or by color:
::
POKI_RUN_COMMAND{{mlr --opprint fraction -f n -g u data/fraction-example.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint fraction -f n -g v data/fraction-example.csv}}HERE
We can see, for example, that 70.9% of females have red (on the left) while 94.5% of reds are for females.
To convert fractions to percents, you may use ``-p``:
::
POKI_RUN_COMMAND{{mlr --opprint fraction -f n -p data/fraction-example.csv}}HERE
Another often-used idiom is to convert from a point distribution to a cumulative distribution, also known as "running sums". Here, you can use ``-c``:
::
POKI_RUN_COMMAND{{mlr --opprint fraction -f n -p -c data/fraction-example.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint fraction -f n -g u -p -c data/fraction-example.csv}}HERE
.. _reference-verbs-grep:
grep
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr grep -h}}HERE
.. _reference-verbs-group-by:
group-by
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr group-by --help}}HERE
This is similar to ``sort`` but with less work. Namely, Miller's sort has three steps: read through the data and append linked lists of records, one for each unique combination of the key-field values; after all records are read, sort the key-field values; then print each record-list. The group-by operation simply omits the middle sort. An example should make this more clear.
::
POKI_RUN_COMMAND{{mlr --opprint group-by a data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint sort -f a data/small}}HERE
In this example, since the sort is on field ``a``, the first step is to group together all records having the same value for field ``a``; the second step is to sort the distinct ``a``-field values ``pan``, ``eks``, and ``wye`` into ``eks``, ``pan``, and ``wye``; the third step is to print out the record-list for ``a=eks``, then the record-list for ``a=pan``, then the record-list for ``a=wye``. The group-by operation omits the middle sort and just puts like records together, for those times when a sort isn't desired. In particular, the ordering of group-by fields for group-by is the order in which they were encountered in the data stream, which in some cases may be more interesting to you.
.. _reference-verbs-group-like:
group-like
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr group-like --help}}HERE
This groups together records having the same schema (i.e. same ordered list of field names) which is useful for making sense of time-ordered output as described in :doc:`record-heterogeneity` -- in particular, in preparation for CSV or pretty-print output.
::
POKI_RUN_COMMAND{{mlr cat data/het.dkvp}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint group-like data/het.dkvp}}HERE
.. _reference-verbs-having-fields:
having-fields
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr having-fields --help}}HERE
Similar to :ref:`reference-verbs-group-like`, this retains records with specified schema.
::
POKI_RUN_COMMAND{{mlr cat data/het.dkvp}}HERE
::
POKI_RUN_COMMAND{{mlr having-fields --at-least resource data/het.dkvp}}HERE
::
POKI_RUN_COMMAND{{mlr having-fields --which-are resource,ok,loadsec data/het.dkvp}}HERE
.. _reference-verbs-head:
head
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr head --help}}HERE
Note that ``head`` is distinct from :ref:`reference-verbs-top` -- ``head`` shows fields which appear first in the data stream; ``top`` shows fields which are numerically largest (or smallest).
::
POKI_RUN_COMMAND{{mlr --opprint head -n 4 data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint head -n 1 -g b data/medium}}HERE
.. _reference-verbs-histogram:
histogram
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr histogram --help}}HERE
This is just a histogram; there's not too much to say here. A note about binning, by example: Suppose you use ``--lo 0.0 --hi 1.0 --nbins 10 -f x``. The input numbers less than 0 or greater than 1 aren't counted in any bin. Input numbers equal to 1 are counted in the last bin. That is, bin 0 has ``0.0 &le; x < 0.1``, bin 1 has ``0.1 &le; x < 0.2``, etc., but bin 9 has ``0.9 &le; x &le; 1.0``.
::
POKI_RUN_COMMAND{{mlr --opprint put '$x2=$x**2;$x3=$x2*$x' then histogram -f x,x2,x3 --lo 0 --hi 1 --nbins 10 data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint put '$x2=$x**2;$x3=$x2*$x' then histogram -f x,x2,x3 --lo 0 --hi 1 --nbins 10 -o my_ data/medium}}HERE
.. _reference-verbs-join:
join
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr join --help}}HERE
Examples:
Join larger table with IDs with smaller ID-to-name lookup table, showing only paired records:
::
POKI_RUN_COMMAND{{mlr --icsvlite --opprint cat data/join-left-example.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --icsvlite --opprint cat data/join-right-example.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --icsvlite --opprint join -u -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv}}HERE
Same, but with sorting the input first:
::
POKI_RUN_COMMAND{{mlr --icsvlite --opprint sort -f idcode then join -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv}}HERE
Same, but showing only unpaired records:
::
POKI_RUN_COMMAND{{mlr --icsvlite --opprint join --np --ul --ur -u -j id -r idcode -f data/join-left-example.csv data/join-right-example.csv}}HERE
Use prefixing options to disambiguate between otherwise identical non-join field names:
::
POKI_RUN_COMMAND{{mlr --csvlite --opprint cat data/self-join.csv data/self-join.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --csvlite --opprint join -j a --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv}}HERE
Use zero join columns:
::
POKI_RUN_COMMAND{{mlr --csvlite --opprint join -j "" --lp left_ --rp right_ -f data/self-join.csv data/self-join.csv}}HERE
.. _reference-verbs-label:
label
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr label --help}}HERE
See also :ref:`reference-verbs-rename`.
Example: Files such as ``/etc/passwd``, ``/etc/group``, and so on have implicit field names which are found in section-5 manpages. These field names may be made explicit as follows:
::
POKI_INCLUDE_ESCAPED(data/label-example.txt)HERE
Likewise, if you have CSV/CSV-lite input data which has somehow been bereft of its header line, you can re-add a header line using ``--implicit-csv-header`` and ``label``:
::
POKI_RUN_COMMAND{{cat data/headerless.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --csv --implicit-csv-header cat data/headerless.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --csv --implicit-csv-header label name,age,status data/headerless.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --icsv --implicit-csv-header --opprint label name,age,status data/headerless.csv}}HERE
.. _reference-verbs-least-frequent:
least-frequent
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr least-frequent -h}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape -n 5}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 -o someothername}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint --from data/colored-shapes.dkvp least-frequent -f shape,color -n 5 -b}}HERE
See also :ref:`reference-verbs-most-frequent`.
.. _reference-verbs-merge-fields:
merge-fields
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr merge-fields --help}}HERE
This is like ``mlr stats1`` but all accumulation is done across fields within each given record: horizontal rather than vertical statistics, if you will.
Examples:
::
POKI_RUN_COMMAND{{mlr --csvlite --opprint cat data/inout.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --csvlite --opprint merge-fields -a min,max,sum -c _in,_out data/inout.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --csvlite --opprint merge-fields -k -a sum -c _in,_out data/inout.csv}}HERE
.. _reference-verbs-most-frequent:
most-frequent
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr most-frequent -h}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape -n 5}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 -o someothername}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint --from data/colored-shapes.dkvp most-frequent -f shape,color -n 5 -b}}HERE
See also :ref:`reference-verbs-least-frequent`.
.. _reference-verbs-nest:
nest
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr nest -h}}HERE
.. _reference-verbs-nothing:
nothing
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr nothing -h}}HERE
.. _reference-verbs-put:
put
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr put --help}}HERE
Features which put shares with filter
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please see the :doc:`reference-dsl` for more information about the expression language for ``mlr put``.
.. _reference-verbs-regularize:
regularize
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr regularize --help}}HERE
This exists since hash-map software in various languages and tools encountered in the wild does not always print similar rows with fields in the same order: ``mlr regularize`` helps clean that up.
See also :ref:`reference-verbs-reorder`.
.. _reference-verbs-remove-empty-columns:
remove-empty-columns
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr remove-empty-columns --help}}HERE
::
POKI_RUN_COMMAND{{cat data/remove-empty-columns.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --csv remove-empty-columns data/remove-empty-columns.csv}}HERE
Since this verb needs to read all records to see if any of them has a non-empty value for a given field name, it is non-streaming: it will ingest all records before writing any.
.. _reference-verbs-rename:
rename
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr rename --help}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint rename i,INDEX,b,COLUMN2 data/small}}HERE
As discussed in :doc:`performance`, ``sed`` is significantly faster than Miller at doing this. However, Miller is format-aware, so it knows to do renames only within specified field keys and not any others, nor in field values which may happen to contain the same pattern. Example:
::
POKI_RUN_COMMAND{{sed 's/y/COLUMN5/g' data/small}}HERE
::
POKI_RUN_COMMAND{{mlr rename y,COLUMN5 data/small}}HERE
See also :ref:`reference-verbs-label`.
.. _reference-verbs-reorder:
reorder
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr reorder --help}}HERE
This pivots specified field names to the start or end of the record -- for
example when you have highly multi-column data and you want to bring a field or
two to the front of line where you can give a quick visual scan.
::
POKI_RUN_COMMAND{{mlr --opprint cat data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint reorder -f i,b data/small}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint reorder -e -f i,b data/small}}HERE
.. _reference-verbs-repeat:
repeat
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr repeat --help}}HERE
This is useful in at least two ways: one, as a data-generator as in the
above example using ``urand()``; two, for reconstructing individual
samples from data which has been count-aggregated:
::
POKI_RUN_COMMAND{{cat data/repeat-example.dat}}HERE
::
POKI_RUN_COMMAND{{mlr repeat -f count then cut -x -f count data/repeat-example.dat}}HERE
After expansion with ``repeat``, such data can then be sent on to
``stats1 -a mode``, or (if the data are numeric) to ``stats1 -a
p10,p50,p90``, etc.
.. _reference-verbs-reshape:
reshape
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr reshape --help}}HERE
.. _reference-verbs-sample:
sample
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr sample --help}}HERE
This is reservoir-sampling: select *k* items from *n* with
uniform probability and no repeats in the sample. (If *n* is less than
*k*, then of course only *n* samples are produced.) With ``-g
{field names}``, produce a *k*-sample for each distinct value of the
specified field names.
::
POKI_INCLUDE_ESCAPED(data/sample-example.txt)HERE
Note that no output is produced until all inputs are in. Another way to do
sampling, which works in the streaming case, is ``mlr filter 'urand() &
0.001'`` where you tune the 0.001 to meet your needs.
.. _reference-verbs-sec2gmt:
sec2gmt
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr sec2gmt -h}}HERE
.. _reference-verbs-sec2gmtdate:
sec2gmtdate
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr sec2gmtdate -h}}HERE
.. _reference-verbs-seqgen:
seqgen
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr seqgen -h}}HERE
::
POKI_RUN_COMMAND{{mlr seqgen --stop 10}}HERE
::
POKI_RUN_COMMAND{{mlr seqgen --start 20 --stop 40 --step 4}}HERE
::
POKI_RUN_COMMAND{{mlr seqgen --start 40 --stop 20 --step -4}}HERE
.. _reference-verbs-shuffle:
shuffle
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr shuffle -h}}HERE
.. _reference-verbs-skip-trivial-records:
skip-trivial-records
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr skip-trivial-records -h}}HERE
::
POKI_RUN_COMMAND{{cat data/trivial-records.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --csv skip-trivial-records data/trivial-records.csv}}HERE
.. _reference-verbs-sort:
sort
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr sort --help}}HERE
Example:
::
POKI_RUN_COMMAND{{mlr --opprint sort -f a -nr x data/small}}HERE
Here's an example filtering log data: suppose multiple threads (labeled here by color) are all logging progress counts to a single log file. The log file is (by nature) chronological, so the progress of various threads is interleaved:
::
POKI_RUN_COMMAND{{head -n 10 data/multicountdown.dat}}HERE
We can group these by thread by sorting on the thread ID (here,
``color``). Since Miller's sort is stable, this means that
timestamps within each thread's log data are still chronological:
::
POKI_RUN_COMMAND{{head -n 20 data/multicountdown.dat | mlr --opprint sort -f color}}HERE
Any records not having all specified sort keys will appear at the end of the output, in the order they
were encountered, regardless of the specified sort order:
::
POKI_RUN_COMMAND{{mlr sort -n x data/sort-missing.dkvp}}HERE
::
POKI_RUN_COMMAND{{mlr sort -nr x data/sort-missing.dkvp}}HERE
.. _reference-verbs-stats1:
stats1
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr stats1 --help}}HERE
These are simple univariate statistics on one or more number-valued fields
(``count`` and ``mode`` apply to non-numeric fields as well),
optionally categorized by one or more other fields.
::
POKI_RUN_COMMAND{{mlr --oxtab stats1 -a count,sum,min,p10,p50,mean,p90,max -f x,y data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint stats1 -a mean -f x,y -g b then sort -f b data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint stats1 -a p50,p99 -f u,v -g color then put '$ur=$u_p99/$u_p50;$vr=$v_p99/$v_p50' data/colored-shapes.dkvp}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint count-distinct -f shape then sort -nr count data/colored-shapes.dkvp}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint stats1 -a mode -f color -g shape data/colored-shapes.dkvp}}HERE
.. _reference-verbs-stats2:
stats2
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr stats2 --help}}HERE
These are simple bivariate statistics on one or more pairs of number-valued
fields, optionally categorized by one or more fields.
::
POKI_RUN_COMMAND{{mlr --oxtab put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' then stats2 -a cov,corr -f x,y,y,y,x2,xy,x2,y2 data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint put '$x2=$x*$x; $xy=$x*$y; $y2=$y**2' then stats2 -a linreg-ols,r2 -f x,y,y,y,xy,y2 -g a data/medium}}HERE
Here's an example simple line-fit. The ``x`` and ``y``
fields of the ``data/medium`` dataset are just independent uniformly
distributed on the unit interval. Here we remove half the data and fit a line to it.
::
POKI_INCLUDE_ESCAPED(data/linreg-example.txt)HERE
I use `pgr <https://github.com/johnkerl/pgr>`_ for plotting; here's a screenshot.
.. image:: data/linreg-example.jpg
(Thanks Drew Kunas for a good conversation about PCA!)
Here's an example estimating time-to-completion for a set of jobs. Input data comes from a log file, with number of work units left to do in the ``count`` field and accumulated seconds in the ``upsec`` field, labeled by the ``color`` field:
::
POKI_RUN_COMMAND{{head -n 10 data/multicountdown.dat}}HERE
We can do a linear regression on count remaining as a function of time: with ``c = m*u+b`` we want to find the time when the count goes to zero, i.e. ``u=-b/m``.
::
POKI_RUN_COMMAND{{mlr --oxtab stats2 -a linreg-pca -f upsec,count -g color then put '$donesec = -$upsec_count_pca_b/$upsec_count_pca_m' data/multicountdown.dat}}HERE
.. _reference-verbs-step:
step
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr step --help}}HERE
Most Miller commands are record-at-a-time, with the exception of ``stats1``, ``stats2``, and ``histogram`` which compute aggregate output. The ``step`` command is intermediate: it allows the option of adding fields which are functions of fields from previous records. Rsum is short for *running sum*.
::
POKI_RUN_COMMAND{{mlr --opprint step -a shift,delta,rsum,counter -f x data/medium | head -15}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint step -a shift,delta,rsum,counter -f x -g a data/medium | head -15}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint step -a ewma -f x -d 0.1,0.9 data/medium | head -15}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint step -a ewma -f x -d 0.1,0.9 -o smooth,rough data/medium | head -15}}HERE
Example deriving uptime-delta from system uptime:
::
POKI_INCLUDE_ESCAPED(data/ping-delta-example.txt)HERE
.. _reference-verbs-tac:
tac
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr tac --help}}HERE
Prints the records in the input stream in reverse order. Note: this requires Miller to retain all input records in memory before any output records are produced.
::
POKI_RUN_COMMAND{{mlr --icsv --opprint cat data/a.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --icsv --opprint cat data/b.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --icsv --opprint tac data/a.csv data/b.csv}}HERE
::
POKI_RUN_COMMAND{{mlr --icsv --opprint put '$filename=FILENAME' then tac data/a.csv data/b.csv}}HERE
.. _reference-verbs-tail:
tail
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr tail --help}}HERE
Prints the last *n* records in the input stream, optionally by category.
::
POKI_RUN_COMMAND{{mlr --opprint tail -n 4 data/colored-shapes.dkvp}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint tail -n 1 -g shape data/colored-shapes.dkvp}}HERE
.. _reference-verbs-tee:
tee
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr tee --help}}HERE
.. _reference-verbs-top:
top
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr top --help}}HERE
Note that ``top`` is distinct from :ref:`reference-verbs-head` -- ``head`` shows fields which appear first in the data stream; ``top`` shows fields which are numerically largest (or smallest).
::
POKI_RUN_COMMAND{{mlr --opprint top -n 4 -f x data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint top -n 4 -f x -o someothername data/medium}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint top -n 2 -f x -g a then sort -f a data/medium}}HERE
.. _reference-verbs-uniq:
uniq
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr uniq --help}}HERE
There are two main ways to use ``mlr uniq``: the first way is with ``-g`` to specify group-by columns.
::
POKI_RUN_COMMAND{{wc -l data/colored-shapes.dkvp}}HERE
::
POKI_RUN_COMMAND{{mlr uniq -g color,shape data/colored-shapes.dkvp}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint uniq -g color,shape -c then sort -f color,shape data/colored-shapes.dkvp}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint uniq -g color,shape -c -o someothername then sort -nr someothername data/colored-shapes.dkvp}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint uniq -n -g color,shape data/colored-shapes.dkvp}}HERE
The second main way to use ``mlr uniq`` is without group-by columns, using ``-a`` instead:
::
POKI_RUN_COMMAND{{cat data/repeats.dkvp}}HERE
::
POKI_RUN_COMMAND{{wc -l data/repeats.dkvp}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint uniq -a data/repeats.dkvp}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint uniq -a -n data/repeats.dkvp}}HERE
::
POKI_RUN_COMMAND{{mlr --opprint uniq -a -c data/repeats.dkvp}}HERE
.. _reference-verbs-unsparsify:
unsparsify
----------------------------------------------------------------
::
POKI_RUN_COMMAND{{mlr unsparsify --help}}HERE
Examples:
::
POKI_RUN_COMMAND{{cat data/sparse.json}}HERE
::
POKI_RUN_COMMAND{{mlr --json unsparsify data/sparse.json}}HERE
::
POKI_RUN_COMMAND{{mlr --ijson --opprint unsparsify data/sparse.json}}HERE
::
POKI_RUN_COMMAND{{mlr --ijson --opprint unsparsify --fill-with missing data/sparse.json}}HERE
Reference in a new issue View git blame Copy permalink