Then-chaining

How do I examine then-chaining?

Then-chaining found in Miller is intended to function the same as Unix pipes, but with less keystroking. You can print your data one pipeline step at a time, to see what intermediate output at one step becomes the input to the next step.

First, look at the input data:

 cat data/then-example.csv
 Status,Payment_Type,Amount
 paid,cash,10.00
 pending,debit,20.00
 paid,cash,50.00
 pending,credit,40.00
 paid,debit,30.00

Next, run the first step of your command, omitting anything from the first then onward:

 mlr --icsv --opprint count-distinct -f Status,Payment_Type data/then-example.csv
 Status  Payment_Type count
 paid    cash         2
 pending debit        1
 pending credit       1
 paid    debit        1

After that, run it with the next then step included:

 mlr --icsv --opprint count-distinct -f Status,Payment_Type \
   then sort -nr count \
   data/then-example.csv
 Status  Payment_Type count
 paid    cash         2
 pending debit        1
 pending credit       1
 paid    debit        1

Now if you use then to include another verb after that, the columns Status, Payment_Type, and count will be the input to that verb.

Note, by the way, that you’ll get the same results using pipes:

 mlr --csv count-distinct -f Status,Payment_Type data/then-example.csv \
 | mlr --icsv --opprint sort -nr count
 Status  Payment_Type count
 paid    cash         2
 pending debit        1
 pending credit       1
 paid    debit        1

NR is not consecutive after then-chaining

Given this input data:

 cat data/small
 a=pan,b=pan,i=1,x=0.3467901443380824,y=0.7268028627434533
 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
 a=wye,b=wye,i=3,x=0.20460330576630303,y=0.33831852551664776
 a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463
 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729

why don’t I see NR=1 and NR=2 here??

 mlr filter '$x > 0.5' then put '$NR = NR' data/small
 a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797,NR=2
 a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729,NR=5

The reason is that NR is computed for the original input records and isn’t dynamically updated. By contrast, NF is dynamically updated: it’s the number of fields in the current record, and if you add/remove a field, the value of NF will change:

 echo x=1,y=2,z=3 | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x,$y,$z; $nf3 = NF'
 nf1=3,u=4,nf2=5,nf3=3

NR, by contrast (and FNR as well), retains the value from the original input stream, and records may be dropped by a filter within a then-chain. To recover consecutive record numbers, you can use out-of-stream variables as follows:

 mlr --opprint --from data/small put '
   begin{ @nr1 = 0 }
   @nr1 += 1;
   $nr1 = @nr1
 ' \
 then filter '$x>0.5' \
 then put '
   begin{ @nr2 = 0 }
   @nr2 += 1;
   $nr2 = @nr2
 '
 a   b   i x                  y                  nr1 nr2
 eks pan 2 0.7586799647899636 0.5221511083334797 2   1
 wye pan 5 0.5732889198020006 0.8636244699032729 5   2

Or, simply use mlr cat -n:

 mlr filter '$x > 0.5' then cat -n data/small
 n=1,a=eks,b=pan,i=2,x=0.7586799647899636,y=0.5221511083334797
 n=2,a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729