# Regular expressions

Miller lets you use regular expressions (of the [types accepted by Go](https://pkg.go.dev/regexp)) in the following contexts:

* In `mlr filter` with `=~` or `!=~`, e.g. `mlr filter '$url =~ "http.*com"'`

* In `mlr put` with `regextract`, e.g. `mlr put '$output = regextract($input, "[a-z][a-z][0-9][0-9]")`

* In `mlr put` with `sub` or `gsub`, e.g. `mlr put '$url = sub($url, "http.*com", "")'`

* In `mlr having-fields`, e.g. `mlr having-fields --any-matching '^sda[0-9]'`

* In `mlr cut`, e.g. `mlr cut -r -f '^status$,^sda[0-9]'`

* In `mlr rename`, e.g. `mlr rename -r '^(sda[0-9]).*$,dev/\1'`

* In `mlr grep`, e.g. `mlr --csv grep 00188555487 myfiles*.csv`

Points demonstrated by the above examples:

* There are no implicit start-of-string or end-of-string anchors; please use `^` and/or `$` explicitly.

* Miller regexes are wrapped with double quotes rather than slashes.

* The `i` after the ending double quote indicates a case-insensitive regex.

* Capture groups are wrapped with `(...)` rather than `\(...\)`; use `\(` and `\)` to match against parentheses.

Example:

GENMD-RUN-COMMAND
cat data/regex-in-data.dat
GENMD-EOF

GENMD-RUN-COMMAND
mlr filter '$name =~ $regex' data/regex-in-data.dat
GENMD-EOF

## Regex captures for the `=~` operator

Regex captures of the form `\0` through `\9` are supported as follows:

* Captures have in-function context for `sub` and `gsub`. For example, the first `\1,\2` pair belong to the first `sub` and the second `\1,\2` pair belong to the second `sub`:

GENMD-SHOW-COMMAND
mlr put '$b = sub($a, "(..)_(...)", "\2-\1"); $c = sub($a, "(..)_(.)(..)", ":\1:\2:\3")'
GENMD-EOF

* Captures endure for the entirety of a `put` for the `=~` and `!=~` operators. For example, here the `\1,\2` are set by the `=~` operator and are used by both subsequent assignment statements:

GENMD-SHOW-COMMAND
mlr put '$a =~ "(..)_(....); $b = "left_\1"; $c = "right_\2"'
GENMD-EOF

* Each user-defined function has its own frame for captures. For example:

GENMD-SHOW-COMMAND
mlr -n put '
func f() {
    if ("456 defg" =~ "([0-9]+) ([a-z]+)") {
        print "INNER: \1 \2";
    }
}
end {
    if ("123 abc" =~ "([0-9]+) ([a-z]+)") {
        print "OUTER PRE:  \1 \2";
        f();
        print "OUTER POST: \1 \2";
    }
}'
GENMD-EOF

* The captures are not retained across multiple puts. For example, here the `\1,\2` won't be expanded from the regex capture:

GENMD-SHOW-COMMAND
mlr put '$a =~ "(..)_(....)' then {... something else ...} then put '$b = "left_\1"; $c = "right_\2"'
GENMD-EOF

* Up to nine matches are supported: `\1` through `\9`, while `\0` is the entire match string; `\15` is treated as `\1` followed by an unrelated `5`.

## Resetting captures

If you use `(...)` in your regular expression, then up to 9 matches are supported for the `=~`
operator, and an arbitrary number of matches are supported for the `match` DSL function.

* Before any match is done, `"\1"` etc. in a string evaluate to themselves.
* After a successful match is done, `"\1"` etc. in a string evaluate to the matched substring.
* After an unsuccessful match is done, `"\1"` etc. in a string evaluate to the empty string.
* You can match against `null` to reset to the original state.

GENMD-CARDIFY-HIGHLIGHT-ONE
mlr repl

[mlr] "\1:\2"
"\1:\2"

[mlr] "abc" =~ "..."
true

[mlr] "\1:\2"
":"

[mlr] "abc" =~ "(.).(.)"
true

[mlr] "\1:\2"
"a:c"

[mlr] "abc" =~ "(.)x(.)"
false

[mlr] "\1:\2"
":"

[mlr] "abc" =~ null

[mlr] "\1:\2"
"\1:\2"
GENMD-EOF

## The `strmatch` and `strmatchx` DSL functions

The `=~` and `!=~` operators have been in Miller for a long time, and they will continue to be
supported.  They do, however, have some deficiencies. As of Miller 6.11 and beyond, the `strmatch`
and `strmatchx` provide more robust ways to do capturing.

First, some examples.

The `strmatch` function only returns a boolean result, and it doesn't set `\0..\9`:

GENMD-CARDIFY-HIGHLIGHT-ONE
mlr repl

[mlr] strmatch("abc", "....")
false

[mlr] strmatch("abc", "...")
true

[mlr] strmatch("abc", "(.).(.)")
true

[mlr] strmatch("[ab:3458]", "([a-z]+):([0-9]+)")
true
GENMD-EOF

The `strmatchx` function also doesn't set `\0..\9`, but returns a map-valued result:

GENMD-CARDIFY-HIGHLIGHT-ONE
mlr repl

[mlr] strmatchx("abc", "....")
{
  "matched": false
}

[mlr] strmatchx("abc", "...")
{
  "matched": true,
  "full_capture": "abc",
  "full_start": 1,
  "full_end": 3
}

[mlr] strmatchx("abc", "(.).(.)")
{
  "matched": true,
  "full_capture": "abc",
  "full_start": 1,
  "full_end": 3,
  "captures": ["a", "c"],
  "starts": [1, 3],
  "ends": [1, 3]
}

[mlr] "[ab:3458]" =~ "([a-z]+):([0-9]+)"
true

[mlr] "\1"
"ab"

[mlr] "\2"
"3458"

[mlr] strmatchx("[ab:3458]", "([a-z]+):([0-9]+)")
{
  "matched": true,
  "full_capture": "ab:3458",
  "full_start": 2,
  "full_end": 8,
  "captures": ["ab", "3458"],
  "starts": [2, 5],
  "ends": [3, 8]
}
GENMD-EOF

Notes:

* When there is no match, the result from `strmatchx` only has the `"matched":false` key/value pair.
* When there is a match with no captures, the result from `strmatchx` has the `"matched":true` key/value pair,
  as well as `full_capture` (taking the place of `\0` set by `=~`), and `full_start` and `full_end`
  which `=~` does not offer.
* When there is a match with no captures, the result from `strmatchx` also has the `captures` array
  whose slots 1, 2, 3, ... are the same as would have been set by `=~` via `\1, \2, \3, ...`.
  However, `strmatchx` offers an arbitrary number of captures, not just `\1..\9`.
  Additionally, the `starts` and `ends` arrays are indices into the input string.
* Since you hold the return value from `strmatchx`, you can operate on it as you wish --- instead of
  relying on the (function-scoped) globals `\0..\9`.
* The price paid is that using `strmatchx` does indeed tend to take more keystrokes than `=~`.

## More information

Regular expressions are those supported by the [Go regexp package](https://pkg.go.dev/regexp), which in turn are of type [RE2](https://github.com/google/re2/wiki/Syntax) except for `\C`:

GENMD-RUN-COMMAND
go doc regexp/syntax
GENMD-EOF

One caveat: for strings in "regex position" -- e.g. the second argument to
[`sub`](reference-dsl-builtin-functions.md#sub) or
[`gsub`](reference-dsl-builtin-functions.md#gsub), or after `=~` -- `"\t"`
means a backslash and a `t` -- which is the right thing -- whereas for strings
in "non-regex position", e.g. anywhere else, `"\t"` becomes the tab character.
This is to say (if you're familiar with r-strings in Python) all strings in
regex position are implicit r-strings.  Generally this is the right thing and
should cause little confusion. Note however that this means `"\t"."\t"` in the
second argument to `sub` isn't the same as `"\t\t"`.
