Document that NF is dynamic within a record (#2192) (#2193)

NF is re-evaluated at each reference, tracking the current field count
of the record as it is modified -- it is not constant per record. This
surprised a user whose C-style for-loop testing i <= NF, with a body
that added fields, became an infinite loop; the docs sentence "their
values ... change from one record to the next" implied per-record
constancy.

Changes:
- reference-dsl-variables.md.in: add an explicit paragraph and live
  example showing NF changing mid-expression as fields are added and
  removed, plus a caution about for-loops bounded by NF whose bodies
  add fields, with the two idiomatic alternatives (snapshot NF first,
  or a key-value for-loop over $* which iterates over a copy).
- reference-dsl-control-structures.md.in: note in the while/do-while
  section that those examples rely on NF's dynamism, and add a
  triple-for note warning about non-terminating NF-bounded loops.
- glossary.md.in: note dynamism in the NF entry.
- pkg/dsl/cst/keyword_usage.go: extend 'mlr help keyword NF' text
  (flows into the manpage and the usage-keywords docs section).
- Also fix a pre-existing typo nearby: "if a field has 5 records" ->
  "if a record has 5 fields".
- Regenerated docs and man pages.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

docs/src/glossary.md

@@ -550,7 +550,11 @@ mathematic constant
 Stands for _number of fields_. A read-only [built-in
 variable](reference-dsl-variables.md#built-in-variables) in the [Miller
 programming language](miller-programming-language.md) which shows the number of fields
-in the current record.
+in the current record. Note that `NF` is dynamic: it is re-evaluated at each
+reference, so if fields are added to or removed from the current record --
+even within a single `put` or `filter` expression -- the value of `NF` changes
+accordingly. See the [built-in-variables
+section](reference-dsl-variables.md#built-in-variables) for details.
 
 ## NIDX
 

docs/src/glossary.md.in

@@ -534,7 +534,11 @@ mathematic constant
 Stands for _number of fields_. A read-only [built-in
 variable](reference-dsl-variables.md#built-in-variables) in the [Miller
 programming language](miller-programming-language.md) which shows the number of fields
-in the current record.
+in the current record. Note that `NF` is dynamic: it is re-evaluated at each
+reference, so if fields are added to or removed from the current record --
+even within a single `put` or `filter` expression -- the value of `NF` changes
+accordingly. See the [built-in-variables
+section](reference-dsl-variables.md#built-in-variables) for details.
 
 ## NIDX
 

docs/src/manpage.md

@@ -4128,7 +4128,10 @@ This is simply a copy of what you should see on running `man mlr` at a command p
        M_PI: the mathematical constant pi.
 
    1mNF0m
-       NF: evaluates to the number of fields in the current record.
+       NF: evaluates to the number of fields in the current record. Note that NF
+       is dynamic: it is re-evaluated at each reference, so if fields are added to or
+       removed from the current record -- even within a single put/filter expression
+       -- the value of NF changes accordingly.
 
    1mNR0m
        NR: evaluates to the number of the current record over all files

docs/src/manpage.txt

@@ -4107,7 +4107,10 @@
        M_PI: the mathematical constant pi.
 
    1mNF0m
-       NF: evaluates to the number of fields in the current record.
+       NF: evaluates to the number of fields in the current record. Note that NF
+       is dynamic: it is re-evaluated at each reference, so if fields are added to or
+       removed from the current record -- even within a single put/filter expression
+       -- the value of NF changes accordingly.
 
    1mNR0m
        NR: evaluates to the number of the current record over all files

docs/src/reference-dsl-control-structures.md

@@ -153,6 +153,11 @@ x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar
 x=1,y=2,3=,4=,5=,foo=bar
 </pre>
 
+Note that these examples rely on the fact that
+[`NF`](reference-dsl-variables.md#built-in-variables) is **dynamic**: it is
+re-evaluated at each reference, so each `$[NF+1] = ""` assignment increments
+`NF`, and the loop makes progress toward its exit condition.
+
 A `break` or `continue` within nested conditional blocks or if-statements will,
 of course, propagate to the innermost loop enclosing them, if any. A `break` or
 `continue` outside a loop is a syntax error that will be flagged as soon as the
@@ -524,6 +529,8 @@ Notes:
 
 * As with all `for`/`if`/`while` statements in Miller, the curly braces are required even if the body is a single statement or empty.
 
+* Since [`NF`](reference-dsl-variables.md#built-in-variables) is **dynamic** -- it is re-evaluated at each reference -- a loop such as `for (i = 1; i <= NF; i += 1)` whose body adds fields to the current record will never terminate, since each added field increments `NF`. To loop over the fields the record had on entry, take a copy first -- e.g. `num n = NF; for (i = 1; i <= n; i += 1) { ... }` -- or use a [key-value for-loop](#key-value-for-loops) such as `for (k, v in $*)`, which iterates over a copy of the record made before the loop began.
+
 ## Begin/end blocks
 
 Miller supports an `awk`-like `begin/end` syntax.  The statements in the `begin` block are executed before any input records are read; the statements in the `end` block are executed after the last input record is read.  (If you want to execute some statement at the start of each file, not at the start of the first file as with `begin`, you might use a pattern/action block of the form `FNR == 1 { ... }`.) All statements outside of `begin` or `end` are, of course, executed on every input record. Semicolons separate statements inside or outside of begin/end blocks; semicolons are required between begin/end block bodies and any subsequent statement.  For example:

docs/src/reference-dsl-control-structures.md.in

@@ -101,6 +101,11 @@ echo x=1,y=2 | mlr put '
 '
 GENMD-EOF
 
+Note that these examples rely on the fact that
+[`NF`](reference-dsl-variables.md#built-in-variables) is **dynamic**: it is
+re-evaluated at each reference, so each `$[NF+1] = ""` assignment increments
+`NF`, and the loop makes progress toward its exit condition.
+
 A `break` or `continue` within nested conditional blocks or if-statements will,
 of course, propagate to the innermost loop enclosing them, if any. A `break` or
 `continue` outside a loop is a syntax error that will be flagged as soon as the
@@ -341,6 +346,8 @@ Notes:
 
 * As with all `for`/`if`/`while` statements in Miller, the curly braces are required even if the body is a single statement or empty.
 
+* Since [`NF`](reference-dsl-variables.md#built-in-variables) is **dynamic** -- it is re-evaluated at each reference -- a loop such as `for (i = 1; i <= NF; i += 1)` whose body adds fields to the current record will never terminate, since each added field increments `NF`. To loop over the fields the record had on entry, take a copy first -- e.g. `num n = NF; for (i = 1; i <= n; i += 1) { ... }` -- or use a [key-value for-loop](#key-value-for-loops) such as `for (k, v in $*)`, which iterates over a copy of the record made before the loop began.
+
 ## Begin/end blocks
 
 Miller supports an `awk`-like `begin/end` syntax.  The statements in the `begin` block are executed before any input records are read; the statements in the `end` block are executed after the last input record is read.  (If you want to execute some statement at the start of each file, not at the start of the first file as with `begin`, you might use a pattern/action block of the form `FNR == 1 { ... }`.) All statements outside of `begin` or `end` are, of course, executed on every input record. Semicolons separate statements inside or outside of begin/end blocks; semicolons are required between begin/end block bodies and any subsequent statement.  For example:

docs/src/reference-dsl-variables.md

@@ -131,7 +131,7 @@ a=eks,b=wye,i=4,x=NEW,y=0.134188
 a=wye,b=pan,i=5,x=0.573288,y=NEW
 </pre>
 
-Right-hand side accesses to non-existent fields -- i.e., with index less than 1 or greater than `NF` -- return an absent value. Likewise, left-hand side accesses only refer to fields that already exist. For example, if a field has 5 records, then assigning the name or value of the 6th (or 600th) field results in a no-op.
+Right-hand side accesses to non-existent fields -- i.e., with index less than 1 or greater than `NF` -- return an absent value. Likewise, left-hand side accesses only refer to fields that already exist. For example, if a record has 5 fields, then assigning the name or value of the 6th (or 600th) field results in a no-op.
 
 <pre class="pre-highlight-in-pair">
 <b>mlr put '$[[6]] = "NEW"' data/small</b>
@@ -608,6 +608,29 @@ system environment variables at the time Miller starts. Any changes made to
 `ENV` by assigning to it will affect any subprocesses, such as using
 [piped tee](reference-dsl-output-statements.md#redirected-output-statements).
 
+Note that, unlike the others, **`NF` is dynamic even within a single record**:
+it is re-evaluated at each reference, tracking the number of fields in the
+current record as of that moment. So if you add or remove fields -- even
+within a single `put` or `filter` expression -- the value of `NF` changes
+accordingly:
+
+<pre class="pre-highlight-in-pair">
+<b>echo 'x=1,y=2,z=3' | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x, $y; $nf3 = NF'</b>
+</pre>
+<pre class="pre-non-highlight-in-pair">
+z=3,nf1=3,u=4,nf2=5,nf3=4
+</pre>
+
+In particular, be careful with loops whose continuation test involves `NF`,
+when the loop body adds fields to the record. For example,
+`for (i = 1; i <= NF; i += 1) { $["copy_".i] = $[[[i]]] }` is an **infinite
+loop**, since each new field increments `NF`, which is re-tested at each
+iteration. To loop over the fields a record had on entry, either take a copy
+of `NF` before the loop -- e.g. `num n = NF; for (i = 1; i <= n; i += 1) {
+... }` -- or use a [key-value for-loop](reference-dsl-control-structures.md#key-value-for-loops)
+such as `for (k, v in $*) { ... }`, which iterates over a copy of the record
+made before the loop began.
+
 Their **scope is global**: you can refer to them in any `filter` or `put` statement. The input-record reader assigns their values:
 
 <pre class="pre-highlight-in-pair">
@@ -1260,7 +1283,10 @@ M_E: the mathematical constant e.
 
 M_PI: the mathematical constant pi.
 
-NF: evaluates to the number of fields in the current record.
+NF: evaluates to the number of fields in the current record. Note that NF
+is dynamic: it is re-evaluated at each reference, so if fields are added to or
+removed from the current record -- even within a single put/filter expression
+-- the value of NF changes accordingly.
 
 NR: evaluates to the number of the current record over all files
 being processed, starting with 1. Does not reset at the start of each file.

docs/src/reference-dsl-variables.md.in

@@ -70,7 +70,7 @@ GENMD-RUN-COMMAND
 mlr put '$[[[NR]]] = "NEW"' data/small
 GENMD-EOF
 
-Right-hand side accesses to non-existent fields -- i.e., with index less than 1 or greater than `NF` -- return an absent value. Likewise, left-hand side accesses only refer to fields that already exist. For example, if a field has 5 records, then assigning the name or value of the 6th (or 600th) field results in a no-op.
+Right-hand side accesses to non-existent fields -- i.e., with index less than 1 or greater than `NF` -- return an absent value. Likewise, left-hand side accesses only refer to fields that already exist. For example, if a record has 5 fields, then assigning the name or value of the 6th (or 600th) field results in a no-op.
 
 GENMD-RUN-COMMAND
 mlr put '$[[6]] = "NEW"' data/small
@@ -320,6 +320,26 @@ system environment variables at the time Miller starts. Any changes made to
 `ENV` by assigning to it will affect any subprocesses, such as using
 [piped tee](reference-dsl-output-statements.md#redirected-output-statements).
 
+Note that, unlike the others, **`NF` is dynamic even within a single record**:
+it is re-evaluated at each reference, tracking the number of fields in the
+current record as of that moment. So if you add or remove fields -- even
+within a single `put` or `filter` expression -- the value of `NF` changes
+accordingly:
+
+GENMD-RUN-COMMAND
+echo 'x=1,y=2,z=3' | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x, $y; $nf3 = NF'
+GENMD-EOF
+
+In particular, be careful with loops whose continuation test involves `NF`,
+when the loop body adds fields to the record. For example,
+`for (i = 1; i <= NF; i += 1) { $["copy_".i] = $[[[i]]] }` is an **infinite
+loop**, since each new field increments `NF`, which is re-tested at each
+iteration. To loop over the fields a record had on entry, either take a copy
+of `NF` before the loop -- e.g. `num n = NF; for (i = 1; i <= n; i += 1) {
+... }` -- or use a [key-value for-loop](reference-dsl-control-structures.md#key-value-for-loops)
+such as `for (k, v in $*) { ... }`, which iterates over a copy of the record
+made before the loop began.
+
 Their **scope is global**: you can refer to them in any `filter` or `put` statement. The input-record reader assigns their values:
 
 GENMD-RUN-COMMAND

man/manpage.txt

@@ -4107,7 +4107,10 @@
        M_PI: the mathematical constant pi.
 
    1mNF0m
-       NF: evaluates to the number of fields in the current record.
+       NF: evaluates to the number of fields in the current record. Note that NF
+       is dynamic: it is re-evaluated at each reference, so if fields are added to or
+       removed from the current record -- even within a single put/filter expression
+       -- the value of NF changes accordingly.
 
    1mNR0m
        NR: evaluates to the number of the current record over all files

man/mlr.1

@@ -6639,7 +6639,10 @@ M_PI: the mathematical constant pi.
 .RS 0
 .\}
 .nf
-NF: evaluates to the number of fields in the current record.
+NF: evaluates to the number of fields in the current record. Note that NF
+is dynamic: it is re-evaluated at each reference, so if fields are added to or
+removed from the current record -- even within a single put/filter expression
+-- the value of NF changes accordingly.
 .fi
 .if n \{\
 .RE

pkg/dsl/cst/keyword_usage.go

@@ -588,7 +588,11 @@ func M_PIKeywordUsage() {
 }
 
 func NFKeywordUsage() {
-	fmt.Println(`evaluates to the number of fields in the current record.`)
+	fmt.Println(
+		`evaluates to the number of fields in the current record. Note that NF
+is dynamic: it is re-evaluated at each reference, so if fields are added to or
+removed from the current record -- even within a single put/filter expression
+-- the value of NF changes accordingly.`)
 }
 
 func NRKeywordUsage() {