mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-21 02:23:20 +00:00
doc neaten
This commit is contained in:
John Kerl
parent
8fb2d6044f
commit
d04555585a
4 changed files with 284 additions and 273 deletions
|
|
@ -54,6 +54,11 @@ MAPVAR CHECKLIST:
|
|||
-> string keys before/after int?
|
||||
-> how to incorporate numeric & lexical flags
|
||||
|
||||
* cookbook x 3:
|
||||
- simple stuff like headerless csv
|
||||
- verbs vs dsl list
|
||||
- weird stuff
|
||||
|
||||
* debt-reduction
|
||||
! naming conventions to clarify transfer/copy semantics
|
||||
- copy/ref of keys vs. values in mlhmmv, local_stack, and callees -- ?!?
|
||||
|
|
|
|||
|
|
@ -228,6 +228,63 @@ stream are passed through to output (written to standard output) unless fields
|
|||
are removed with <tt>cut</tt>, or records are excluded with <tt>filter</tt> or
|
||||
<tt>put -q</tt>, etc. Simply assign a value to a field and it will be output.
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h2>Out-of-stream variables</h2>
|
||||
|
||||
<p/> These are prefixed with an at-sign, e.g. <tt>@sum</tt>. Furthermore,
|
||||
unlike built-in variables and stream-record fields, they are maintained in an
|
||||
arbitrarily nested hashmap: you can do <tt>@sum += $quanity</tt>, or
|
||||
<tt>@sum[$color] += $quanity</tt>, or <tt>@sum[$color][$shape] +=
|
||||
$quanity</tt>. The keys for the multi-level hashmap can be any expression which
|
||||
evaluates to string or integer: e.g. <tt>@sum[NR] = $a + $b</tt>,
|
||||
<tt>@sum[$a."-".$b] = $x</tt>, etc.
|
||||
|
||||
<p/> Their names and their values are entirely under your control; they change
|
||||
only when you assign to them.
|
||||
|
||||
<p/> Just as for field names in stream records, if you want to define out-of-stream variables
|
||||
with <b>special characters</b> such as <tt>.</tt> then you can use braces, e.g. <tt>'@{variable.name}["index"]'</tt>.
|
||||
|
||||
<p/>You may use a <b>computed key </b> in square brackets, e.g.
|
||||
|
||||
POKI_RUN_COMMAND{{echo s=green,t=blue,a=3,b=4 | mlr put -q '@[$s."_".$t] = $a * $b; emit all'}}HERE
|
||||
|
||||
<p/> Out-of-stream variables are <b>scoped</b> to the <tt>put</tt> command in
|
||||
which they appear. In particular, if you have two or more <tt>put</tt>
|
||||
commands separated by <tt>then</tt>, each put will have its own set of
|
||||
out-of-stream variables:
|
||||
|
||||
POKI_RUN_COMMAND{{cat data/a.dkvp}}HERE
|
||||
POKI_RUN_COMMAND{{mlr put '@sum += $a; end {emit @sum}' then put 'ispresent($a) {$a=10*$a; @sum += $a}; end {emit @sum}' data/a.dkvp}}HERE
|
||||
|
||||
<p/> Out-of-stream variables’ <b>extent</b> is from the start to the end of the record stream,
|
||||
i.e. every time the <tt>put</tt> or <tt>filter</tt> statement referring to them is executed.
|
||||
|
||||
<p/> Out-of-stream variables are <b>read-write</b>: you can do <tt>$sum=@sum</tt>, <tt>@sum=$sum</tt>,
|
||||
etc.
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h2>Indexed out-of-stream variables</h2>
|
||||
|
||||
<p/>Using an index on the <tt>@count</tt> and <tt>@sum</tt> variables, we get the benefit of the
|
||||
<tt>-g</tt> (group-by) option which <tt>mlr stats1</tt> and various other Miller commands have:
|
||||
|
||||
POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-6.sh)HERE
|
||||
POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-7.sh)HERE
|
||||
|
||||
<p/>Indices can be arbitrarily deep — here there are two or more of them:
|
||||
|
||||
POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-6a.sh)HERE
|
||||
|
||||
The idea is that <tt>stats1</tt>, and other Miller commands, encapsulate
|
||||
frequently-used patterns with a minimum of keystroking (and run a little
|
||||
faster), whereas using out-of-stream variables you have more flexibility and
|
||||
control in what you do.
|
||||
|
||||
<p/>Begin/end blocks can be mixed with pattern/action blocks. For example:
|
||||
|
||||
POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-8.sh)HERE
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h2>Local variables</h2>
|
||||
|
||||
|
|
@ -475,60 +532,6 @@ when the function is called:
|
|||
</pre>
|
||||
</div>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h2>Out-of-stream variables</h2>
|
||||
|
||||
<p/> These are prefixed with an at-sign, e.g. <tt>@sum</tt>. Furthermore,
|
||||
unlike built-in variables and stream-record fields, they are maintained in an
|
||||
arbitrarily nested hashmap: you can do <tt>@sum += $quanity</tt>, or
|
||||
<tt>@sum[$color] += $quanity</tt>, or <tt>@sum[$color][$shape] +=
|
||||
$quanity</tt>. The keys for the multi-level hashmap can be any expression which
|
||||
evaluates to string or integer: e.g. <tt>@sum[NR] = $a + $b</tt>,
|
||||
<tt>@sum[$a."-".$b] = $x</tt>, etc.
|
||||
|
||||
<p/> Their names and their values are entirely under your control; they change
|
||||
only when you assign to them.
|
||||
|
||||
<p/> Just as for field names in stream records, if you want to define out-of-stream variables
|
||||
with <b>special characters</b> such as <tt>.</tt> then you can use braces, e.g. <tt>'@{variable.name}["index"]'</tt>.
|
||||
|
||||
<p/>You may use a <b>computed key </b> in square brackets, e.g.
|
||||
|
||||
POKI_RUN_COMMAND{{echo s=green,t=blue,a=3,b=4 | mlr put -q '@[$s."_".$t] = $a * $b; emit all'}}HERE
|
||||
|
||||
<p/> Out-of-stream variables are <b>scoped</b> to the <tt>put</tt> command in
|
||||
which they appear. In particular, if you have two or more <tt>put</tt>
|
||||
commands separated by <tt>then</tt>, each put will have its own set of
|
||||
out-of-stream variables:
|
||||
|
||||
POKI_RUN_COMMAND{{cat data/a.dkvp}}HERE
|
||||
POKI_RUN_COMMAND{{mlr put '@sum += $a; end {emit @sum}' then put 'ispresent($a) {$a=10*$a; @sum += $a}; end {emit @sum}' data/a.dkvp}}HERE
|
||||
|
||||
<p/> Out-of-stream variables are read-write: you can do <tt>$sum=@sum</tt>, <tt>@sum=$sum</tt>,
|
||||
etc.
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h2>Indexed out-of-stream variables</h2>
|
||||
|
||||
<p/>Using an index on the <tt>@count</tt> and <tt>@sum</tt> variables, we get the benefit of the
|
||||
<tt>-g</tt> (group-by) option which <tt>mlr stats1</tt> and various other Miller commands have:
|
||||
|
||||
POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-6.sh)HERE
|
||||
POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-7.sh)HERE
|
||||
|
||||
<p/>Indices can be arbitrarily deep — here there are two or more of them:
|
||||
|
||||
POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-6a.sh)HERE
|
||||
|
||||
The idea is that <tt>stats1</tt>, and other Miller commands, encapsulate
|
||||
frequently-used patterns with a minimum of keystroking (and run a little
|
||||
faster), whereas using out-of-stream variables you have more flexibility and
|
||||
control in what you do.
|
||||
|
||||
<p/>Begin/end blocks can be mixed with pattern/action blocks. For example:
|
||||
|
||||
POKI_INCLUDE_AND_RUN_ESCAPED(data/begin-end-example-8.sh)HERE
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<h2>Aggregate variable assignments</h2>
|
||||
|
||||
|
|
|
|||
|
|
@ -607,33 +607,33 @@ $ mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
|
|||
' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds
|
||||
i o fcount seconds_delta
|
||||
1 1 1 0
|
||||
2 2 3 0.000028133
|
||||
3 3 5 0.000013828
|
||||
4 5 9 0.000017166
|
||||
5 8 15 0.000022888
|
||||
6 13 25 0.000035048
|
||||
7 21 41 0.000047922
|
||||
8 34 67 0.000075102
|
||||
9 55 109 0.000117064
|
||||
10 89 177 0.000185966
|
||||
11 144 287 0.000295877
|
||||
12 233 465 0.000476122
|
||||
13 377 753 0.001272917
|
||||
14 610 1219 0.001202106
|
||||
15 987 1973 0.001879930
|
||||
16 1597 3193 0.003035069
|
||||
17 2584 5167 0.005100012
|
||||
18 4181 8361 0.009974003
|
||||
19 6765 13529 0.012595892
|
||||
20 10946 21891 0.019485950
|
||||
21 17711 35421 0.031538010
|
||||
22 28657 57313 0.050611019
|
||||
23 46368 92735 0.084169149
|
||||
24 75025 150049 0.131798983
|
||||
25 121393 242785 0.211344004
|
||||
26 196418 392835 0.345608950
|
||||
27 317811 635621 0.557806015
|
||||
28 514229 1028457 0.899939060
|
||||
2 2 3 0.000025988
|
||||
3 3 5 0.000013113
|
||||
4 5 9 0.000015974
|
||||
5 8 15 0.000021935
|
||||
6 13 25 0.000031948
|
||||
7 21 41 0.000046015
|
||||
8 34 67 0.000072002
|
||||
9 55 109 0.000109911
|
||||
10 89 177 0.000177145
|
||||
11 144 287 0.000280857
|
||||
12 233 465 0.000450134
|
||||
13 377 753 0.000726938
|
||||
14 610 1219 0.001173019
|
||||
15 987 1973 0.001877069
|
||||
16 1597 3193 0.003116846
|
||||
17 2584 5167 0.004981041
|
||||
18 4181 8361 0.007852077
|
||||
19 6765 13529 0.012398958
|
||||
20 10946 21891 0.020217896
|
||||
21 17711 35421 0.039513111
|
||||
22 28657 57313 0.053396940
|
||||
23 46368 92735 0.089246035
|
||||
24 75025 150049 0.142396927
|
||||
25 121393 242785 0.235119104
|
||||
26 196418 392835 0.355266094
|
||||
27 317811 635621 0.571523905
|
||||
28 514229 1028457 0.946005106
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
|
|
@ -665,33 +665,33 @@ $ mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put '
|
|||
' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds
|
||||
i o fcount seconds_delta
|
||||
1 1 1 0
|
||||
2 2 3 0.000028133
|
||||
3 3 3 0.000012875
|
||||
4 5 3 0.000010967
|
||||
2 2 3 0.000069141
|
||||
3 3 3 0.000022888
|
||||
4 5 3 0.000012159
|
||||
5 8 3 0.000010967
|
||||
6 13 3 0.000010014
|
||||
6 13 3 0.000011921
|
||||
7 21 3 0.000010014
|
||||
8 34 3 0.000010967
|
||||
9 55 3 0.000010014
|
||||
10 89 3 0.000010014
|
||||
10 89 3 0.000010967
|
||||
11 144 3 0.000010014
|
||||
12 233 3 0.000014067
|
||||
13 377 3 0.000010014
|
||||
12 233 3 0.000015020
|
||||
13 377 3 0.000011921
|
||||
14 610 3 0.000010014
|
||||
15 987 3 0.000010967
|
||||
16 1597 3 0.000010014
|
||||
17 2584 3 0.000010014
|
||||
18 4181 3 0.000010014
|
||||
18 4181 3 0.000010967
|
||||
19 6765 3 0.000010014
|
||||
20 10946 3 0.000009060
|
||||
21 17711 3 0.000011921
|
||||
20 10946 3 0.000010014
|
||||
21 17711 3 0.000010014
|
||||
22 28657 3 0.000010014
|
||||
23 46368 3 0.000011921
|
||||
23 46368 3 0.000013113
|
||||
24 75025 3 0.000010967
|
||||
25 121393 3 0.000010014
|
||||
26 196418 3 0.000010014
|
||||
27 317811 3 0.000010967
|
||||
28 514229 3 0.000010014
|
||||
25 121393 3 0.000011921
|
||||
26 196418 3 0.000010967
|
||||
27 317811 3 0.000010014
|
||||
28 514229 3 0.000010967
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
|
|
|
|||
|
|
@ -151,12 +151,12 @@ Miller commands were run with pretty-print-tabular output format.
|
|||
• <a href="#Variables">Variables</a><br/>
|
||||
• <a href="#Built-in_variables">Built-in variables</a><br/>
|
||||
• <a href="#Field_names">Field names</a><br/>
|
||||
• <a href="#Out-of-stream_variables">Out-of-stream variables</a><br/>
|
||||
• <a href="#Indexed_out-of-stream_variables">Indexed out-of-stream variables</a><br/>
|
||||
• <a href="#Local_variables">Local variables</a><br/>
|
||||
• <a href="#Type-checking">Type-checking</a><br/>
|
||||
• <a href="#Type-test_and_type-assertion_expressions">Type-test and type-assertion expressions</a><br/>
|
||||
• <a href="#Type-declarations_for_local_variables,_function_parameter,_and_function_return_values">Type-declarations for local variables, function parameter, and function return values</a><br/>
|
||||
• <a href="#Out-of-stream_variables">Out-of-stream variables</a><br/>
|
||||
• <a href="#Indexed_out-of-stream_variables">Indexed out-of-stream variables</a><br/>
|
||||
• <a href="#Aggregate_variable_assignments">Aggregate variable assignments</a><br/>
|
||||
• <a href="#Keywords_for_filter_and_put">Keywords for filter and put</a><br/>
|
||||
• <a href="#Operator_precedence">Operator precedence</a><br/>
|
||||
|
|
@ -672,6 +672,184 @@ stream are passed through to output (written to standard output) unless fields
|
|||
are removed with <tt>cut</tt>, or records are excluded with <tt>filter</tt> or
|
||||
<tt>put -q</tt>, etc. Simply assign a value to a field and it will be output.
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="Out-of-stream_variables"/><h2>Out-of-stream variables</h2>
|
||||
|
||||
<p/> These are prefixed with an at-sign, e.g. <tt>@sum</tt>. Furthermore,
|
||||
unlike built-in variables and stream-record fields, they are maintained in an
|
||||
arbitrarily nested hashmap: you can do <tt>@sum += $quanity</tt>, or
|
||||
<tt>@sum[$color] += $quanity</tt>, or <tt>@sum[$color][$shape] +=
|
||||
$quanity</tt>. The keys for the multi-level hashmap can be any expression which
|
||||
evaluates to string or integer: e.g. <tt>@sum[NR] = $a + $b</tt>,
|
||||
<tt>@sum[$a."-".$b] = $x</tt>, etc.
|
||||
|
||||
<p/> Their names and their values are entirely under your control; they change
|
||||
only when you assign to them.
|
||||
|
||||
<p/> Just as for field names in stream records, if you want to define out-of-stream variables
|
||||
with <b>special characters</b> such as <tt>.</tt> then you can use braces, e.g. <tt>'@{variable.name}["index"]'</tt>.
|
||||
|
||||
<p/>You may use a <b>computed key </b> in square brackets, e.g.
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ echo s=green,t=blue,a=3,b=4 | mlr put -q '@[$s."_".$t] = $a * $b; emit all'
|
||||
green_blue=12
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
|
||||
<p/> Out-of-stream variables are <b>scoped</b> to the <tt>put</tt> command in
|
||||
which they appear. In particular, if you have two or more <tt>put</tt>
|
||||
commands separated by <tt>then</tt>, each put will have its own set of
|
||||
out-of-stream variables:
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ cat data/a.dkvp
|
||||
a=1,b=2,c=3
|
||||
a=4,b=5,c=6
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr put '@sum += $a; end {emit @sum}' then put 'ispresent($a) {$a=10*$a; @sum += $a}; end {emit @sum}' data/a.dkvp
|
||||
a=10,b=2,c=3
|
||||
a=40,b=5,c=6
|
||||
sum=5
|
||||
sum=50
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
|
||||
<p/> Out-of-stream variables’ <b>extent</b> is from the start to the end of the record stream,
|
||||
i.e. every time the <tt>put</tt> or <tt>filter</tt> statement referring to them is executed.
|
||||
|
||||
<p/> Out-of-stream variables are <b>read-write</b>: you can do <tt>$sum=@sum</tt>, <tt>@sum=$sum</tt>,
|
||||
etc.
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="Indexed_out-of-stream_variables"/><h2>Indexed out-of-stream variables</h2>
|
||||
|
||||
<p/>Using an index on the <tt>@count</tt> and <tt>@sum</tt> variables, we get the benefit of the
|
||||
<tt>-g</tt> (group-by) option which <tt>mlr stats1</tt> and various other Miller commands have:
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr put -q '
|
||||
@x_count[$a] += 1;
|
||||
@x_sum[$a] += $x;
|
||||
end {
|
||||
emit @x_count, "a";
|
||||
emit @x_sum, "a";
|
||||
}
|
||||
' ../data/small
|
||||
a=pan,x_count=2
|
||||
a=eks,x_count=3
|
||||
a=wye,x_count=2
|
||||
a=zee,x_count=2
|
||||
a=hat,x_count=1
|
||||
a=pan,x_sum=0.849416
|
||||
a=eks,x_sum=1.751863
|
||||
a=wye,x_sum=0.777892
|
||||
a=zee,x_sum=1.125680
|
||||
a=hat,x_sum=0.031442
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr stats1 -a count,sum -f x -g a ../data/small
|
||||
a=pan,x_count=2,x_sum=0.849416
|
||||
a=eks,x_count=3,x_sum=1.751863
|
||||
a=wye,x_count=2,x_sum=0.777892
|
||||
a=zee,x_count=2,x_sum=1.125680
|
||||
a=hat,x_count=1,x_sum=0.031442
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
|
||||
<p/>Indices can be arbitrarily deep — here there are two or more of them:
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr --from data/medium put -q '
|
||||
@x_count[$a][$b] += 1;
|
||||
@x_sum[$a][$b] += $x;
|
||||
end {
|
||||
emit (@x_count, @x_sum), "a", "b";
|
||||
}
|
||||
'
|
||||
a=pan,b=pan,x_count=427,x_sum=219.185129
|
||||
a=pan,b=wye,x_count=395,x_sum=198.432931
|
||||
a=pan,b=eks,x_count=429,x_sum=216.075228
|
||||
a=pan,b=hat,x_count=417,x_sum=205.222776
|
||||
a=pan,b=zee,x_count=413,x_sum=205.097518
|
||||
a=eks,b=pan,x_count=371,x_sum=179.963030
|
||||
a=eks,b=wye,x_count=407,x_sum=196.945286
|
||||
a=eks,b=zee,x_count=357,x_sum=176.880365
|
||||
a=eks,b=eks,x_count=413,x_sum=215.916097
|
||||
a=eks,b=hat,x_count=417,x_sum=208.783171
|
||||
a=wye,b=wye,x_count=377,x_sum=185.295850
|
||||
a=wye,b=pan,x_count=392,x_sum=195.847900
|
||||
a=wye,b=hat,x_count=426,x_sum=212.033183
|
||||
a=wye,b=zee,x_count=385,x_sum=194.774048
|
||||
a=wye,b=eks,x_count=386,x_sum=204.812961
|
||||
a=zee,b=pan,x_count=389,x_sum=202.213804
|
||||
a=zee,b=wye,x_count=455,x_sum=233.991394
|
||||
a=zee,b=eks,x_count=391,x_sum=190.961778
|
||||
a=zee,b=zee,x_count=403,x_sum=206.640635
|
||||
a=zee,b=hat,x_count=409,x_sum=191.300006
|
||||
a=hat,b=wye,x_count=423,x_sum=208.883010
|
||||
a=hat,b=zee,x_count=385,x_sum=196.349450
|
||||
a=hat,b=eks,x_count=389,x_sum=189.006793
|
||||
a=hat,b=hat,x_count=381,x_sum=182.853532
|
||||
a=hat,b=pan,x_count=363,x_sum=168.553807
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
|
||||
The idea is that <tt>stats1</tt>, and other Miller commands, encapsulate
|
||||
frequently-used patterns with a minimum of keystroking (and run a little
|
||||
faster), whereas using out-of-stream variables you have more flexibility and
|
||||
control in what you do.
|
||||
|
||||
<p/>Begin/end blocks can be mixed with pattern/action blocks. For example:
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr put '
|
||||
begin {
|
||||
@num_total = 0;
|
||||
@num_positive = 0;
|
||||
};
|
||||
@num_total += 1;
|
||||
$x > 0.0 {
|
||||
@num_positive += 1;
|
||||
$y = log10($x); $z = sqrt($y)
|
||||
};
|
||||
end {
|
||||
emitf @num_total, @num_positive
|
||||
}
|
||||
' data/put-gating-example-1.dkvp
|
||||
x=-1
|
||||
x=0
|
||||
x=1,y=0.000000,z=0.000000
|
||||
x=2,y=0.301030,z=0.548662
|
||||
x=3,y=0.477121,z=0.690740
|
||||
num_total=5,num_positive=3
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="Local_variables"/><h2>Local variables</h2>
|
||||
|
||||
|
|
@ -1169,181 +1347,6 @@ when the function is called:
|
|||
</pre>
|
||||
</div>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="Out-of-stream_variables"/><h2>Out-of-stream variables</h2>
|
||||
|
||||
<p/> These are prefixed with an at-sign, e.g. <tt>@sum</tt>. Furthermore,
|
||||
unlike built-in variables and stream-record fields, they are maintained in an
|
||||
arbitrarily nested hashmap: you can do <tt>@sum += $quanity</tt>, or
|
||||
<tt>@sum[$color] += $quanity</tt>, or <tt>@sum[$color][$shape] +=
|
||||
$quanity</tt>. The keys for the multi-level hashmap can be any expression which
|
||||
evaluates to string or integer: e.g. <tt>@sum[NR] = $a + $b</tt>,
|
||||
<tt>@sum[$a."-".$b] = $x</tt>, etc.
|
||||
|
||||
<p/> Their names and their values are entirely under your control; they change
|
||||
only when you assign to them.
|
||||
|
||||
<p/> Just as for field names in stream records, if you want to define out-of-stream variables
|
||||
with <b>special characters</b> such as <tt>.</tt> then you can use braces, e.g. <tt>'@{variable.name}["index"]'</tt>.
|
||||
|
||||
<p/>You may use a <b>computed key </b> in square brackets, e.g.
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ echo s=green,t=blue,a=3,b=4 | mlr put -q '@[$s."_".$t] = $a * $b; emit all'
|
||||
green_blue=12
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
|
||||
<p/> Out-of-stream variables are <b>scoped</b> to the <tt>put</tt> command in
|
||||
which they appear. In particular, if you have two or more <tt>put</tt>
|
||||
commands separated by <tt>then</tt>, each put will have its own set of
|
||||
out-of-stream variables:
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ cat data/a.dkvp
|
||||
a=1,b=2,c=3
|
||||
a=4,b=5,c=6
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr put '@sum += $a; end {emit @sum}' then put 'ispresent($a) {$a=10*$a; @sum += $a}; end {emit @sum}' data/a.dkvp
|
||||
a=10,b=2,c=3
|
||||
a=40,b=5,c=6
|
||||
sum=5
|
||||
sum=50
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
|
||||
<p/> Out-of-stream variables are read-write: you can do <tt>$sum=@sum</tt>, <tt>@sum=$sum</tt>,
|
||||
etc.
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="Indexed_out-of-stream_variables"/><h2>Indexed out-of-stream variables</h2>
|
||||
|
||||
<p/>Using an index on the <tt>@count</tt> and <tt>@sum</tt> variables, we get the benefit of the
|
||||
<tt>-g</tt> (group-by) option which <tt>mlr stats1</tt> and various other Miller commands have:
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr put -q '
|
||||
@x_count[$a] += 1;
|
||||
@x_sum[$a] += $x;
|
||||
end {
|
||||
emit @x_count, "a";
|
||||
emit @x_sum, "a";
|
||||
}
|
||||
' ../data/small
|
||||
a=pan,x_count=2
|
||||
a=eks,x_count=3
|
||||
a=wye,x_count=2
|
||||
a=zee,x_count=2
|
||||
a=hat,x_count=1
|
||||
a=pan,x_sum=0.849416
|
||||
a=eks,x_sum=1.751863
|
||||
a=wye,x_sum=0.777892
|
||||
a=zee,x_sum=1.125680
|
||||
a=hat,x_sum=0.031442
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr stats1 -a count,sum -f x -g a ../data/small
|
||||
a=pan,x_count=2,x_sum=0.849416
|
||||
a=eks,x_count=3,x_sum=1.751863
|
||||
a=wye,x_count=2,x_sum=0.777892
|
||||
a=zee,x_count=2,x_sum=1.125680
|
||||
a=hat,x_count=1,x_sum=0.031442
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
|
||||
<p/>Indices can be arbitrarily deep — here there are two or more of them:
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr --from data/medium put -q '
|
||||
@x_count[$a][$b] += 1;
|
||||
@x_sum[$a][$b] += $x;
|
||||
end {
|
||||
emit (@x_count, @x_sum), "a", "b";
|
||||
}
|
||||
'
|
||||
a=pan,b=pan,x_count=427,x_sum=219.185129
|
||||
a=pan,b=wye,x_count=395,x_sum=198.432931
|
||||
a=pan,b=eks,x_count=429,x_sum=216.075228
|
||||
a=pan,b=hat,x_count=417,x_sum=205.222776
|
||||
a=pan,b=zee,x_count=413,x_sum=205.097518
|
||||
a=eks,b=pan,x_count=371,x_sum=179.963030
|
||||
a=eks,b=wye,x_count=407,x_sum=196.945286
|
||||
a=eks,b=zee,x_count=357,x_sum=176.880365
|
||||
a=eks,b=eks,x_count=413,x_sum=215.916097
|
||||
a=eks,b=hat,x_count=417,x_sum=208.783171
|
||||
a=wye,b=wye,x_count=377,x_sum=185.295850
|
||||
a=wye,b=pan,x_count=392,x_sum=195.847900
|
||||
a=wye,b=hat,x_count=426,x_sum=212.033183
|
||||
a=wye,b=zee,x_count=385,x_sum=194.774048
|
||||
a=wye,b=eks,x_count=386,x_sum=204.812961
|
||||
a=zee,b=pan,x_count=389,x_sum=202.213804
|
||||
a=zee,b=wye,x_count=455,x_sum=233.991394
|
||||
a=zee,b=eks,x_count=391,x_sum=190.961778
|
||||
a=zee,b=zee,x_count=403,x_sum=206.640635
|
||||
a=zee,b=hat,x_count=409,x_sum=191.300006
|
||||
a=hat,b=wye,x_count=423,x_sum=208.883010
|
||||
a=hat,b=zee,x_count=385,x_sum=196.349450
|
||||
a=hat,b=eks,x_count=389,x_sum=189.006793
|
||||
a=hat,b=hat,x_count=381,x_sum=182.853532
|
||||
a=hat,b=pan,x_count=363,x_sum=168.553807
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
|
||||
The idea is that <tt>stats1</tt>, and other Miller commands, encapsulate
|
||||
frequently-used patterns with a minimum of keystroking (and run a little
|
||||
faster), whereas using out-of-stream variables you have more flexibility and
|
||||
control in what you do.
|
||||
|
||||
<p/>Begin/end blocks can be mixed with pattern/action blocks. For example:
|
||||
|
||||
<p/>
|
||||
<div class="pokipanel">
|
||||
<pre>
|
||||
$ mlr put '
|
||||
begin {
|
||||
@num_total = 0;
|
||||
@num_positive = 0;
|
||||
};
|
||||
@num_total += 1;
|
||||
$x > 0.0 {
|
||||
@num_positive += 1;
|
||||
$y = log10($x); $z = sqrt($y)
|
||||
};
|
||||
end {
|
||||
emitf @num_total, @num_positive
|
||||
}
|
||||
' data/put-gating-example-1.dkvp
|
||||
x=-1
|
||||
x=0
|
||||
x=1,y=0.000000,z=0.000000
|
||||
x=2,y=0.301030,z=0.548662
|
||||
x=3,y=0.477121,z=0.690740
|
||||
num_total=5,num_positive=3
|
||||
</pre>
|
||||
</div>
|
||||
<p/>
|
||||
|
||||
<!-- ================================================================ -->
|
||||
<a id="Aggregate_variable_assignments"/><h2>Aggregate variable assignments</h2>
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue