DSL reference: output statements

You can output variable-values or expressions in five ways:

  • Assign them to stream-record fields. For example, $cumulative_sum = @sum. For another example, $nr = NR adds a field named nr to each output record, containing the value of the built-in variable NR as of when that record was ingested.

  • Use the print or eprint keywords which immediately print an expression directly to standard output or standard error, respectively. Note that dump, edump, print, and eprint don’t output records which participate in then-chaining; rather, they’re just immediate prints to stdout/stderr. The printn and eprintn keywords are the same except that they don’t print final newlines. Additionally, you can print to a specified file instead of stdout/stderr.

  • Use the dump or edump keywords, which immediately print all out-of-stream variables as a JSON data structure to the standard output or standard error (respectively).

  • Use tee which formats the current stream record (not just an arbitrary string as with print) to a specific file.

  • Use emit/emitp/emitf to send out-of-stream variables’ current values to the output record stream, e.g. @sum += $x; emit @sum which produces an extra output record such as sum=3.1648382.

For the first two options you are populating the output-records stream which feeds into the next verb in a then-chain (if any), or which otherwise is formatted for output using --o... flags.

For the last three options you are sending output directly to standard output, standard error, or a file.

Dump statements

The dump statement is for printing expressions, including maps, directly to stdout/stderr, respectively:

  • There are two variants: dump prints to stdout; edump prints to stderr.

  • Output goes directly to stdout/stderr, respectively: data produced this way do not go downstream to the next verb in a then-chain. (Use emit for that.)

  • You can use dump to output single strings, numbers, or expressions including map-valued data. Map-valued data are printed as JSON. Miller allows string and integer keys in its map literals while JSON allows only string keys, so use mlr put --jknquoteint if you want integer-valued map keys not double-quoted.

  • If you use dump (or edump) with no arguments, you get a JSON structure representing the current values of all out-of-stream variables.

  • As with print, you can redirect output to files.

  • See also Redirected-output statements for examples.

Tee statements

Records produced by a mlr put go downstream to the next verb in your then-chain, if any, or otherwise to standard output. If you want to additionally copy out records to files, you can do that using tee.

The syntax is, by example, mlr --from myfile.dat put 'tee > "tap.dat", $*' then sort -n index. First is tee >, then the filename expression (which can be an expression such as "tap.".$a.".dat"), then a comma, then $*. (Nothing else but $* is teeable.)

See also Redirected-output statements for examples.

Redirected-output statements

The print, dump tee, emitf, emit, and emitp keywords all allow you to redirect output to one or more files or pipe-to commands. The filenames/commands are strings which can be constructed using record-dependent values, so you can do things like splitting a table into multiple files, one for each account ID, and so on.

Details:

  • The print and dump keywords produce output immediately to standard output, or to specified file(s) or pipe-to command if present.

 mlr help keyword print
 print: prints expression immediately to stdout.

   Example: mlr --from f.dat put -q 'print "The sum of x and y is ".($x+$y)'
   Example: mlr --from f.dat put -q 'for (k, v in $*) { print k . " => " . v }'
   Example: mlr --from f.dat put  '(NR %% 1000 == 0) { print > stderr, "Checkpoint ".NR}'

 mlr help keyword dump
 dump: prints all currently defined out-of-stream variables immediately
 to stdout as JSON.

 With >, >>, or |, the data do not become part of the output record stream but
 are instead redirected.

 The > and >> are for write and append, as in the shell, but (as with awk) the
 file-overwrite for > is on first write, not per record. The | is for piping to
 a process which will process the data. There will be one open file for each
 distinct file name (for > and >>) or one subordinate process for each distinct
 value of the piped-to command (for |). Output-formatting flags are taken from
 the main command line.

   Example: mlr --from f.dat put -q '@v[NR]=$*; end { dump }'
   Example: mlr --from f.dat put -q '@v[NR]=$*; end { dump >  "mytap.dat"}'
   Example: mlr --from f.dat put -q '@v[NR]=$*; end { dump >> "mytap.dat"}'
   Example: mlr --from f.dat put -q '@v[NR]=$*; end { dump | "jq .[]"}'

  • mlr put sends the current record (possibly modified by the put expression) to the output record stream. Records are then input to the following verb in a then-chain (if any), else printed to standard output (unless put -q). The tee keyword additionally writes the output record to specified file(s) or pipe-to command, or immediately to stdout/stderr.

 mlr help keyword tee
 tee: prints the current record to specified file.
 This is an immediate print to the specified file (except for pprint format
 which of course waits until the end of the input stream to format all output).

 The > and >> are for write and append, as in the shell, but (as with awk) the
 file-overwrite for > is on first write, not per record. The | is for piping to
 a process which will process the data. There will be one open file for each
 distinct file name (for > and >>) or one subordinate process for each distinct
 value of the piped-to command (for |). Output-formatting flags are taken from
 the main command line.

 You can use any of the output-format command-line flags, e.g. --ocsv, --ofs,
 etc., to control the format of the output. See also mlr -h.

 emit with redirect and tee with redirect are identical, except tee can only
 output $*.

   Example: mlr --from f.dat put 'tee >  "/tmp/data-".$a, $*'
   Example: mlr --from f.dat put 'tee >> "/tmp/data-".$a.$b, $*'
   Example: mlr --from f.dat put 'tee >  stderr, $*'
   Example: mlr --from f.dat put -q 'tee | "tr \[a-z\\] \[A-Z\\]", $*'
   Example: mlr --from f.dat put -q 'tee | "tr \[a-z\\] \[A-Z\\] > /tmp/data-".$a, $*'
   Example: mlr --from f.dat put -q 'tee | "gzip > /tmp/data-".$a.".gz", $*'
   Example: mlr --from f.dat put -q --ojson 'tee | "gzip > /tmp/data-".$a.".gz", $*'

  • mlr put’s emitf, emitp, and emit send out-of-stream variables to the output record stream. These are then input to the following verb in a then-chain (if any), else printed to standard output. When redirected with >, >>, or |, they instead write the out-of-stream variable(s) to specified file(s) or pipe-to command, or immediately to stdout/stderr.

 mlr help keyword emitf
 emitf: inserts non-indexed out-of-stream variable(s) side-by-side into the
 output record stream.

 With >, >>, or |, the data do not become part of the output record stream but
 are instead redirected.

 The > and >> are for write and append, as in the shell, but (as with awk) the
 file-overwrite for > is on first write, not per record. The | is for piping to
 a process which will process the data. There will be one open file for each
 distinct file name (for > and >>) or one subordinate process for each distinct
 value of the piped-to command (for |). Output-formatting flags are taken from
 the main command line.

 You can use any of the output-format command-line flags, e.g. --ocsv, --ofs,
 etc., to control the format of the output if the output is redirected. See also mlr -h.

   Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf @a'
   Example: mlr --from f.dat put --oxtab '@a=$i;@b+=$x;@c+=$y; emitf > "tap-".$i.".dat", @a'
   Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf @a, @b, @c'
   Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf > "mytap.dat", @a, @b, @c'
   Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf >> "mytap.dat", @a, @b, @c'
   Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf > stderr, @a, @b, @c'
   Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf | "grep somepattern", @a, @b, @c'
   Example: mlr --from f.dat put '@a=$i;@b+=$x;@c+=$y; emitf | "grep somepattern > mytap.dat", @a, @b, @c'

 Please see https://johnkerl.org/miller6://johnkerl.org/miller/doc for more information.

 mlr help keyword emitp
 emitp: inserts an out-of-stream variable into the output record stream.
 Hashmap indices present in the data but not slotted by emitp arguments are
 output concatenated with ":".

 With >, >>, or |, the data do not become part of the output record stream but
 are instead redirected.

 The > and >> are for write and append, as in the shell, but (as with awk) the
 file-overwrite for > is on first write, not per record. The | is for piping to
 a process which will process the data. There will be one open file for each
 distinct file name (for > and >>) or one subordinate process for each distinct
 value of the piped-to command (for |). Output-formatting flags are taken from
 the main command line.

 You can use any of the output-format command-line flags, e.g. --ocsv, --ofs,
 etc., to control the format of the output if the output is redirected. See also mlr -h.

   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp @sums'
   Example: mlr --from f.dat put --opprint '@sums[$a][$b]+=$x; emitp > "tap-".$a.$b.".dat", @sums'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp @sums, "index1", "index2"'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp @*, "index1", "index2"'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp >  "mytap.dat", @*, "index1", "index2"'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp >> "mytap.dat", @*, "index1", "index2"'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp | "gzip > mytap.dat.gz", @*, "index1", "index2"'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp > stderr, @*, "index1", "index2"'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emitp | "grep somepattern", @*, "index1", "index2"'

 Please see https://johnkerl.org/miller6://johnkerl.org/miller/doc for more information.

 mlr help keyword emit
 emit: inserts an out-of-stream variable into the output record stream. Hashmap
 indices present in the data but not slotted by emit arguments are not output.

 With >, >>, or |, the data do not become part of the output record stream but
 are instead redirected.

 The > and >> are for write and append, as in the shell, but (as with awk) the
 file-overwrite for > is on first write, not per record. The | is for piping to
 a process which will process the data. There will be one open file for each
 distinct file name (for > and >>) or one subordinate process for each distinct
 value of the piped-to command (for |). Output-formatting flags are taken from
 the main command line.

 You can use any of the output-format command-line flags, e.g. --ocsv, --ofs,
 etc., to control the format of the output if the output is redirected. See also mlr -h.

   Example: mlr --from f.dat put 'emit >  "/tmp/data-".$a, $*'
   Example: mlr --from f.dat put 'emit >  "/tmp/data-".$a, mapexcept($*, "a")'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit @sums'
   Example: mlr --from f.dat put --ojson '@sums[$a][$b]+=$x; emit > "tap-".$a.$b.".dat", @sums'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit @sums, "index1", "index2"'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit @*, "index1", "index2"'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit >  "mytap.dat", @*, "index1", "index2"'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit >> "mytap.dat", @*, "index1", "index2"'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit | "gzip > mytap.dat.gz", @*, "index1", "index2"'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit > stderr, @*, "index1", "index2"'
   Example: mlr --from f.dat put '@sums[$a][$b]+=$x; emit | "grep somepattern", @*, "index1", "index2"'

 Please see https://johnkerl.org/miller6://johnkerl.org/miller/doc for more information.

Emit statements

There are three variants: emitf, emit, and emitp. Keep in mind that out-of-stream variables are a nested, multi-level hashmap (directly viewable as JSON using dump), whereas Miller output records are lists of single-level key-value pairs. The three emit variants allow you to control how the multilevel hashmaps are flatten down to output records. You can emit any map-valued expression, including $*, map-valued out-of-stream variables, the entire out-of-stream-variable collection @*, map-valued local variables, map literals, or map-valued function return values.

Use emitf to output several out-of-stream variables side-by-side in the same output record. For emitf these mustn’t have indexing using @name[...]. Example:

 mlr put -q '
   @count += 1;
   @x_sum += $x;
   @y_sum += $y;
   end { emitf @count, @x_sum, @y_sum}
 ' data/small
 count=5,x_sum=2.264761728567491,y_sum=2.585085709781158

Use emit to output an out-of-stream variable. If it’s non-indexed you’ll get a simple key-value pair:

 cat data/small
 a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
 a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729

 mlr put -q '@sum += $x; end { dump }' data/small
 {
   "sum": 2.264761728567491
 }

 mlr put -q '@sum += $x; end { emit @sum }' data/small
 sum=2.264761728567491

If it’s indexed then use as many names after emit as there are indices:

 mlr put -q '@sum[$a] += $x; end { dump }' data/small
 {
   "sum": {
     "pan": 0.3467901443380824,
     "eks": 1.1400793586611044,
     "wye": 0.7778922255683036
   }
 }

 mlr put -q '@sum[$a] += $x; end { emit @sum, "a" }' data/small
 a=pan,sum=0.3467901443380824
 a=eks,sum=1.1400793586611044
 a=wye,sum=0.7778922255683036

 mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small
 {
   "sum": {
     "pan": {
       "pan": 0.3467901443380824
     },
     "eks": {
       "pan": 0.7586799647899636,
       "wye": 0.38139939387114097
     },
     "wye": {
       "wye": 0.20460330576630303,
       "pan": 0.5732889198020006
     }
   }
 }

 mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a", "b" }' data/small
 a=pan,b=pan,sum=0.3467901443380824
 a=eks,b=pan,sum=0.7586799647899636
 a=eks,b=wye,sum=0.38139939387114097
 a=wye,b=wye,sum=0.20460330576630303
 a=wye,b=pan,sum=0.5732889198020006

 mlr put -q '@sum[$a][$b][$i] += $x; end { dump }' data/small
 {
   "sum": {
     "pan": {
       "pan": {
         "1": 0.3467901443380824
       }
     },
     "eks": {
       "pan": {
         "2": 0.7586799647899636
       },
       "wye": {
         "4": 0.38139939387114097
       }
     },
     "wye": {
       "wye": {
         "3": 0.20460330576630303
       },
       "pan": {
         "5": 0.5732889198020006
       }
     }
   }
 }

 mlr put -q '
   @sum[$a][$b][$i] += $x;
   end { emit @sum, "a", "b", "i" }
 ' data/small
 a=pan,b=pan,i=1,sum=0.3467901443380824
 a=eks,b=pan,i=2,sum=0.7586799647899636
 a=eks,b=wye,i=4,sum=0.38139939387114097
 a=wye,b=wye,i=3,sum=0.20460330576630303
 a=wye,b=pan,i=5,sum=0.5732889198020006

Now for emitp: if you have as many names following emit as there are levels in the out-of-stream variable’s hashmap, then emit and emitp do the same thing. Where they differ is when you don’t specify as many names as there are hashmap levels. In this case, Miller needs to flatten multiple map indices down to output-record keys: emitp includes full prefixing (hence the p in emitp) while emit takes the deepest hashmap key as the output-record key:

 mlr put -q '@sum[$a][$b] += $x; end { dump }' data/small
 {
   "sum": {
     "pan": {
       "pan": 0.3467901443380824
     },
     "eks": {
       "pan": 0.7586799647899636,
       "wye": 0.38139939387114097
     },
     "wye": {
       "wye": 0.20460330576630303,
       "pan": 0.5732889198020006
     }
   }
 }

 mlr put -q '@sum[$a][$b] += $x; end { emit @sum, "a" }' data/small
 a=pan,pan=0.3467901443380824
 a=eks,pan=0.7586799647899636,wye=0.38139939387114097
 a=wye,wye=0.20460330576630303,pan=0.5732889198020006

 mlr put -q '@sum[$a][$b] += $x; end { emit @sum }' data/small
 pan.pan=0.3467901443380824,eks.pan=0.7586799647899636,eks.wye=0.38139939387114097,wye.wye=0.20460330576630303,wye.pan=0.5732889198020006

 mlr put -q '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small
 a=pan,sum.pan=0.3467901443380824
 a=eks,sum.pan=0.7586799647899636,sum.wye=0.38139939387114097
 a=wye,sum.wye=0.20460330576630303,sum.pan=0.5732889198020006

 mlr put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small
 sum.pan.pan=0.3467901443380824,sum.eks.pan=0.7586799647899636,sum.eks.wye=0.38139939387114097,sum.wye.wye=0.20460330576630303,sum.wye.pan=0.5732889198020006

 mlr --oxtab put -q '@sum[$a][$b] += $x; end { emitp @sum }' data/small
 sum.pan.pan 0.3467901443380824
 sum.eks.pan 0.7586799647899636
 sum.eks.wye 0.38139939387114097
 sum.wye.wye 0.20460330576630303
 sum.wye.pan 0.5732889198020006

Use –oflatsep to specify the character which joins multilevel keys for emitp (it defaults to a colon):

 mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum, "a" }' data/small
 a=pan,sum.pan=0.3467901443380824
 a=eks,sum.pan=0.7586799647899636,sum.wye=0.38139939387114097
 a=wye,sum.wye=0.20460330576630303,sum.pan=0.5732889198020006

 mlr put -q --oflatsep / '@sum[$a][$b] += $x; end { emitp @sum }' data/small
 sum.pan.pan=0.3467901443380824,sum.eks.pan=0.7586799647899636,sum.eks.wye=0.38139939387114097,sum.wye.wye=0.20460330576630303,sum.wye.pan=0.5732889198020006

 mlr --oxtab put -q --oflatsep / '
   @sum[$a][$b] += $x;
   end { emitp @sum }
 ' data/small
 sum.pan.pan 0.3467901443380824
 sum.eks.pan 0.7586799647899636
 sum.eks.wye 0.38139939387114097
 sum.wye.wye 0.20460330576630303
 sum.wye.pan 0.5732889198020006

Multi-emit statements

You can emit multiple map-valued expressions side-by-side by including their names in parentheses:

 mlr --from data/medium --opprint put -q '
   @x_count[$a][$b] += 1;
   @x_sum[$a][$b] += $x;
   end {
       for ((a, b), _ in @x_count) {
           @x_mean[a][b] = @x_sum[a][b] / @x_count[a][b]
       }
       emit (@x_sum, @x_count, @x_mean), "a", "b"
   }
 '
 a   b   x_sum              x_count x_mean
 pan pan 219.1851288316854  427     0.5133141190437597
 pan wye 198.43293070748447 395     0.5023618498923658
 pan eks 216.07522773165525 429     0.5036718595143479
 pan hat 205.22277621488686 417     0.492140950155604
 pan zee 205.09751802331917 413     0.4966041598627583
 eks pan 179.96303047250723 371     0.48507555383425127
 eks wye 196.9452860713734  407     0.4838950517724162
 eks zee 176.8803651584733  357     0.49546320772681596
 eks eks 215.91609712937984 413     0.5227992666570941
 eks hat 208.783170520597   417     0.5006790659966355
 wye wye 185.29584980261419 377     0.49150092785839306
 wye pan 195.84790012056564 392     0.4996119901034838
 wye hat 212.0331829346132  426     0.4977304763723314
 wye zee 194.77404756708714 385     0.5059066170573692
 wye eks 204.8129608356315  386     0.5306035254809106
 zee pan 202.21380378504267 389     0.5198298297816007
 zee wye 233.9913939194868  455     0.5142667998230479
 zee eks 190.9617780631925  391     0.4883932942792647
 zee zee 206.64063510417319 403     0.5127559183726382
 zee hat 191.30000620900935 409     0.46772617655014515
 hat wye 208.8830097609959  423     0.49381326184632596
 hat zee 196.3494502965293  385     0.5099985721987774
 hat eks 189.0067933716193  389     0.48587864619953547
 hat hat 182.8535323148762  381     0.47993053101017374
 hat pan 168.5538067327806  363     0.4643355557376876

What this does is walk through the first out-of-stream variable (@x_sum in this example) as usual, then for each keylist found (e.g. pan,wye), include the values for the remaining out-of-stream variables (here, @x_count and @x_mean). You should use this when all out-of-stream variables in the emit statement have the same shape and the same keylists.

Emit-all statements

Use emit all (or emit @* which is synonymous) to output all out-of-stream variables. You can use the following idiom to get various accumulators output side-by-side (reminiscent of mlr stats1):

 mlr --from data/small --opprint put -q '
   @v[$a][$b]["sum"] += $x;
   @v[$a][$b]["count"] += 1;
   end{emit @*,"a","b"}
 '
 a b   pan.sum            pan.count
 v pan 0.3467901443380824 1

 a b   pan.sum            pan.count wye.sum             wye.count
 v eks 0.7586799647899636 1         0.38139939387114097 1

 a b   wye.sum             wye.count pan.sum            pan.count
 v wye 0.20460330576630303 1         0.5732889198020006 1

 mlr --from data/small --opprint put -q '
   @sum[$a][$b] += $x;
   @count[$a][$b] += 1;
   end{emit @*,"a","b"}
 '
 a   b   pan
 sum pan 0.3467901443380824

 a   b   pan                wye
 sum eks 0.7586799647899636 0.38139939387114097

 a   b   wye                 pan
 sum wye 0.20460330576630303 0.5732889198020006

 a     b   pan
 count pan 1

 a     b   pan wye
 count eks 1   1

 a     b   wye pan
 count wye 1   1

 mlr --from data/small --opprint put -q '
   @sum[$a][$b] += $x;
   @count[$a][$b] += 1;
   end{emit (@sum, @count),"a","b"}
 '
 a   b   sum                 count
 pan pan 0.3467901443380824  1
 eks pan 0.7586799647899636  1
 eks wye 0.38139939387114097 1
 wye wye 0.20460330576630303 1
 wye pan 0.5732889198020006  1