diff --git a/docs/src/glossary.md b/docs/src/glossary.md index 5875d2a47..9dce6e454 100644 --- a/docs/src/glossary.md +++ b/docs/src/glossary.md @@ -550,7 +550,11 @@ mathematic constant Stands for _number of fields_. A read-only [built-in variable](reference-dsl-variables.md#built-in-variables) in the [Miller programming language](miller-programming-language.md) which shows the number of fields -in the current record. +in the current record. Note that `NF` is dynamic: it is re-evaluated at each +reference, so if fields are added to or removed from the current record -- +even within a single `put` or `filter` expression -- the value of `NF` changes +accordingly. See the [built-in-variables +section](reference-dsl-variables.md#built-in-variables) for details. ## NIDX diff --git a/docs/src/glossary.md.in b/docs/src/glossary.md.in index 4002cae75..6bcb5c7af 100644 --- a/docs/src/glossary.md.in +++ b/docs/src/glossary.md.in @@ -534,7 +534,11 @@ mathematic constant Stands for _number of fields_. A read-only [built-in variable](reference-dsl-variables.md#built-in-variables) in the [Miller programming language](miller-programming-language.md) which shows the number of fields -in the current record. +in the current record. Note that `NF` is dynamic: it is re-evaluated at each +reference, so if fields are added to or removed from the current record -- +even within a single `put` or `filter` expression -- the value of `NF` changes +accordingly. See the [built-in-variables +section](reference-dsl-variables.md#built-in-variables) for details. ## NIDX diff --git a/docs/src/manpage.md b/docs/src/manpage.md index 023bb0f3c..5c216d509 100644 --- a/docs/src/manpage.md +++ b/docs/src/manpage.md @@ -4128,7 +4128,10 @@ This is simply a copy of what you should see on running `man mlr` at a command p M_PI: the mathematical constant pi. 1mNF0m - NF: evaluates to the number of fields in the current record. + NF: evaluates to the number of fields in the current record. Note that NF + is dynamic: it is re-evaluated at each reference, so if fields are added to or + removed from the current record -- even within a single put/filter expression + -- the value of NF changes accordingly. 1mNR0m NR: evaluates to the number of the current record over all files diff --git a/docs/src/manpage.txt b/docs/src/manpage.txt index 161f7f0c6..b235fea97 100644 --- a/docs/src/manpage.txt +++ b/docs/src/manpage.txt @@ -4107,7 +4107,10 @@ M_PI: the mathematical constant pi. 1mNF0m - NF: evaluates to the number of fields in the current record. + NF: evaluates to the number of fields in the current record. Note that NF + is dynamic: it is re-evaluated at each reference, so if fields are added to or + removed from the current record -- even within a single put/filter expression + -- the value of NF changes accordingly. 1mNR0m NR: evaluates to the number of the current record over all files diff --git a/docs/src/reference-dsl-control-structures.md b/docs/src/reference-dsl-control-structures.md index fd4c4d977..d6148390d 100644 --- a/docs/src/reference-dsl-control-structures.md +++ b/docs/src/reference-dsl-control-structures.md @@ -153,6 +153,11 @@ x=1,y=2,3=,4=,5=,6=,7=,8=,9=,10=,foo=bar x=1,y=2,3=,4=,5=,foo=bar +Note that these examples rely on the fact that +[`NF`](reference-dsl-variables.md#built-in-variables) is **dynamic**: it is +re-evaluated at each reference, so each `$[NF+1] = ""` assignment increments +`NF`, and the loop makes progress toward its exit condition. + A `break` or `continue` within nested conditional blocks or if-statements will, of course, propagate to the innermost loop enclosing them, if any. A `break` or `continue` outside a loop is a syntax error that will be flagged as soon as the @@ -524,6 +529,8 @@ Notes: * As with all `for`/`if`/`while` statements in Miller, the curly braces are required even if the body is a single statement or empty. +* Since [`NF`](reference-dsl-variables.md#built-in-variables) is **dynamic** -- it is re-evaluated at each reference -- a loop such as `for (i = 1; i <= NF; i += 1)` whose body adds fields to the current record will never terminate, since each added field increments `NF`. To loop over the fields the record had on entry, take a copy first -- e.g. `num n = NF; for (i = 1; i <= n; i += 1) { ... }` -- or use a [key-value for-loop](#key-value-for-loops) such as `for (k, v in $*)`, which iterates over a copy of the record made before the loop began. + ## Begin/end blocks Miller supports an `awk`-like `begin/end` syntax. The statements in the `begin` block are executed before any input records are read; the statements in the `end` block are executed after the last input record is read. (If you want to execute some statement at the start of each file, not at the start of the first file as with `begin`, you might use a pattern/action block of the form `FNR == 1 { ... }`.) All statements outside of `begin` or `end` are, of course, executed on every input record. Semicolons separate statements inside or outside of begin/end blocks; semicolons are required between begin/end block bodies and any subsequent statement. For example: diff --git a/docs/src/reference-dsl-control-structures.md.in b/docs/src/reference-dsl-control-structures.md.in index caffa9bdf..ecb39c3d5 100644 --- a/docs/src/reference-dsl-control-structures.md.in +++ b/docs/src/reference-dsl-control-structures.md.in @@ -101,6 +101,11 @@ echo x=1,y=2 | mlr put ' ' GENMD-EOF +Note that these examples rely on the fact that +[`NF`](reference-dsl-variables.md#built-in-variables) is **dynamic**: it is +re-evaluated at each reference, so each `$[NF+1] = ""` assignment increments +`NF`, and the loop makes progress toward its exit condition. + A `break` or `continue` within nested conditional blocks or if-statements will, of course, propagate to the innermost loop enclosing them, if any. A `break` or `continue` outside a loop is a syntax error that will be flagged as soon as the @@ -341,6 +346,8 @@ Notes: * As with all `for`/`if`/`while` statements in Miller, the curly braces are required even if the body is a single statement or empty. +* Since [`NF`](reference-dsl-variables.md#built-in-variables) is **dynamic** -- it is re-evaluated at each reference -- a loop such as `for (i = 1; i <= NF; i += 1)` whose body adds fields to the current record will never terminate, since each added field increments `NF`. To loop over the fields the record had on entry, take a copy first -- e.g. `num n = NF; for (i = 1; i <= n; i += 1) { ... }` -- or use a [key-value for-loop](#key-value-for-loops) such as `for (k, v in $*)`, which iterates over a copy of the record made before the loop began. + ## Begin/end blocks Miller supports an `awk`-like `begin/end` syntax. The statements in the `begin` block are executed before any input records are read; the statements in the `end` block are executed after the last input record is read. (If you want to execute some statement at the start of each file, not at the start of the first file as with `begin`, you might use a pattern/action block of the form `FNR == 1 { ... }`.) All statements outside of `begin` or `end` are, of course, executed on every input record. Semicolons separate statements inside or outside of begin/end blocks; semicolons are required between begin/end block bodies and any subsequent statement. For example: diff --git a/docs/src/reference-dsl-variables.md b/docs/src/reference-dsl-variables.md index 0d8c296c1..84b7ca2d5 100644 --- a/docs/src/reference-dsl-variables.md +++ b/docs/src/reference-dsl-variables.md @@ -131,7 +131,7 @@ a=eks,b=wye,i=4,x=NEW,y=0.134188 a=wye,b=pan,i=5,x=0.573288,y=NEW -Right-hand side accesses to non-existent fields -- i.e., with index less than 1 or greater than `NF` -- return an absent value. Likewise, left-hand side accesses only refer to fields that already exist. For example, if a field has 5 records, then assigning the name or value of the 6th (or 600th) field results in a no-op. +Right-hand side accesses to non-existent fields -- i.e., with index less than 1 or greater than `NF` -- return an absent value. Likewise, left-hand side accesses only refer to fields that already exist. For example, if a record has 5 fields, then assigning the name or value of the 6th (or 600th) field results in a no-op. 
 mlr put '$[[6]] = "NEW"' data/small
@@ -608,6 +608,29 @@ system environment variables at the time Miller starts. Any changes made to
 `ENV` by assigning to it will affect any subprocesses, such as using
 [piped tee](reference-dsl-output-statements.md#redirected-output-statements).
 
+Note that, unlike the others, **`NF` is dynamic even within a single record**:
+it is re-evaluated at each reference, tracking the number of fields in the
+current record as of that moment. So if you add or remove fields -- even
+within a single `put` or `filter` expression -- the value of `NF` changes
+accordingly:
+
+
+echo 'x=1,y=2,z=3' | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x, $y; $nf3 = NF'
+

+
+z=3,nf1=3,u=4,nf2=5,nf3=4
+

+
+In particular, be careful with loops whose continuation test involves `NF`,
+when the loop body adds fields to the record. For example,
+`for (i = 1; i <= NF; i += 1) { $["copy_".i] = $[[[i]]] }` is an **infinite
+loop**, since each new field increments `NF`, which is re-tested at each
+iteration. To loop over the fields a record had on entry, either take a copy
+of `NF` before the loop -- e.g. `num n = NF; for (i = 1; i <= n; i += 1) {
+... }` -- or use a [key-value for-loop](reference-dsl-control-structures.md#key-value-for-loops)
+such as `for (k, v in $*) { ... }`, which iterates over a copy of the record
+made before the loop began.
+
 Their **scope is global**: you can refer to them in any `filter` or `put` statement. The input-record reader assigns their values:
 
 
@@ -1260,7 +1283,10 @@ M_E: the mathematical constant e.
 
 M_PI: the mathematical constant pi.
 
-NF: evaluates to the number of fields in the current record.
+NF: evaluates to the number of fields in the current record. Note that NF
+is dynamic: it is re-evaluated at each reference, so if fields are added to or
+removed from the current record -- even within a single put/filter expression
+-- the value of NF changes accordingly.
 
 NR: evaluates to the number of the current record over all files
 being processed, starting with 1. Does not reset at the start of each file.
diff --git a/docs/src/reference-dsl-variables.md.in b/docs/src/reference-dsl-variables.md.in
index 0b9ddf60b..47e6549f3 100644
--- a/docs/src/reference-dsl-variables.md.in
+++ b/docs/src/reference-dsl-variables.md.in
@@ -70,7 +70,7 @@ GENMD-RUN-COMMAND
 mlr put '$[[[NR]]] = "NEW"' data/small
 GENMD-EOF
 
-Right-hand side accesses to non-existent fields -- i.e., with index less than 1 or greater than `NF` -- return an absent value. Likewise, left-hand side accesses only refer to fields that already exist. For example, if a field has 5 records, then assigning the name or value of the 6th (or 600th) field results in a no-op.
+Right-hand side accesses to non-existent fields -- i.e., with index less than 1 or greater than `NF` -- return an absent value. Likewise, left-hand side accesses only refer to fields that already exist. For example, if a record has 5 fields, then assigning the name or value of the 6th (or 600th) field results in a no-op.
 
 GENMD-RUN-COMMAND
 mlr put '$[[6]] = "NEW"' data/small
@@ -320,6 +320,26 @@ system environment variables at the time Miller starts. Any changes made to
 `ENV` by assigning to it will affect any subprocesses, such as using
 [piped tee](reference-dsl-output-statements.md#redirected-output-statements).
 
+Note that, unlike the others, **`NF` is dynamic even within a single record**:
+it is re-evaluated at each reference, tracking the number of fields in the
+current record as of that moment. So if you add or remove fields -- even
+within a single `put` or `filter` expression -- the value of `NF` changes
+accordingly:
+
+GENMD-RUN-COMMAND
+echo 'x=1,y=2,z=3' | mlr put '$nf1 = NF; $u = 4; $nf2 = NF; unset $x, $y; $nf3 = NF'
+GENMD-EOF
+
+In particular, be careful with loops whose continuation test involves `NF`,
+when the loop body adds fields to the record. For example,
+`for (i = 1; i <= NF; i += 1) { $["copy_".i] = $[[[i]]] }` is an **infinite
+loop**, since each new field increments `NF`, which is re-tested at each
+iteration. To loop over the fields a record had on entry, either take a copy
+of `NF` before the loop -- e.g. `num n = NF; for (i = 1; i <= n; i += 1) {
+... }` -- or use a [key-value for-loop](reference-dsl-control-structures.md#key-value-for-loops)
+such as `for (k, v in $*) { ... }`, which iterates over a copy of the record
+made before the loop began.
+
 Their **scope is global**: you can refer to them in any `filter` or `put` statement. The input-record reader assigns their values:
 
 GENMD-RUN-COMMAND
diff --git a/man/manpage.txt b/man/manpage.txt
index 161f7f0c6..b235fea97 100644
--- a/man/manpage.txt
+++ b/man/manpage.txt
@@ -4107,7 +4107,10 @@
        M_PI: the mathematical constant pi.
 
    1mNF0m
-       NF: evaluates to the number of fields in the current record.
+       NF: evaluates to the number of fields in the current record. Note that NF
+       is dynamic: it is re-evaluated at each reference, so if fields are added to or
+       removed from the current record -- even within a single put/filter expression
+       -- the value of NF changes accordingly.
 
    1mNR0m
        NR: evaluates to the number of the current record over all files
diff --git a/man/mlr.1 b/man/mlr.1
index 0ff2c2ad1..3c5b28c52 100644
--- a/man/mlr.1
+++ b/man/mlr.1
@@ -6639,7 +6639,10 @@ M_PI: the mathematical constant pi.
 .RS 0
 .\}
 .nf
-NF: evaluates to the number of fields in the current record.
+NF: evaluates to the number of fields in the current record. Note that NF
+is dynamic: it is re-evaluated at each reference, so if fields are added to or
+removed from the current record -- even within a single put/filter expression
+-- the value of NF changes accordingly.
 .fi
 .if n \{\
 .RE
diff --git a/pkg/dsl/cst/keyword_usage.go b/pkg/dsl/cst/keyword_usage.go
index 43e2e1c7c..b4fbdcce7 100644
--- a/pkg/dsl/cst/keyword_usage.go
+++ b/pkg/dsl/cst/keyword_usage.go
@@ -588,7 +588,11 @@ func M_PIKeywordUsage() {
 }
 
 func NFKeywordUsage() {
-	fmt.Println(`evaluates to the number of fields in the current record.`)
+	fmt.Println(
+		`evaluates to the number of fields in the current record. Note that NF
+is dynamic: it is re-evaluated at each reference, so if fields are added to or
+removed from the current record -- even within a single put/filter expression
+-- the value of NF changes accordingly.`)
 }
 
 func NRKeywordUsage() {