diff --git a/docs/src/glossary.md b/docs/src/glossary.md index d71abf1d6..bb731297b 100644 --- a/docs/src/glossary.md +++ b/docs/src/glossary.md @@ -769,6 +769,13 @@ can split it into several files, one for each distinct `id`. See the [section on tee statements](reference-dsl-output-statements.md#tee-statements) for an example. +## terminals + +These include `mlr help`, `mlr regtest`, `mlr repl`, and `mlr version`. They +aren't verbs but they can be preceded by various command-line flags. They're in +contrast to [auxents](#auxents) which are effectively standalone programs +packaged with Miller. + ## terminator Used in two senses: diff --git a/docs/src/manpage.md b/docs/src/manpage.md index 33325ce9d..65ffe0d8c 100644 --- a/docs/src/manpage.md +++ b/docs/src/manpage.md @@ -141,6 +141,7 @@ MILLER(1) MILLER(1) mlr help file-formats Flags: mlr help flags + mlr help flag mlr help list-separator-aliases mlr help list-separator-regex-aliases mlr help comments-in-data-flags @@ -173,6 +174,7 @@ MILLER(1) MILLER(1) mlr help keyword Other: mlr help auxents + mlr help terminals mlr help mlrrc mlr help output-colorization mlr help type-arithmetic-info @@ -801,16 +803,12 @@ MILLER(1) MILLER(1) --rs {string} Specify RS for input and output. 1mAUXILIARY COMMANDS0m - Available subcommands: - aux-list - hex - lecat - termcvt - unhex - help - regtest - repl - version + Available entries: + mlr aux-list + mlr hex + mlr lecat + mlr termcvt + mlr unhex For more information, please invoke mlr {subcommand} --help. 1mMLRRC0m @@ -1203,17 +1201,18 @@ MILLER(1) MILLER(1) Options: -i Use case-insensitive search. -v Invert: pass through records which do not match the regex. + -a Only grep for values, not keys and values. -h|--help Show this message. Note that "mlr filter" is more powerful, but requires you to know field names. - By contrast, "mlr grep" allows you to regex-match the entire record. It does - this by formatting each record in memory as DKVP, using command-line-specified - ORS/OFS/OPS, and matching the resulting line against the regex specified - here. In particular, the regex is not applied to the input stream: if you - have CSV with header line "x,y,z" and data line "1,2,3" then the regex will - be matched, not against either of these lines, but against the DKVP line - "x=1,y=2,z=3". Furthermore, not all the options to system grep are supported, - and this command is intended to be merely a keystroke-saver. To get all the - features of system grep, you can do + By contrast, "mlr grep" allows you to regex-match the entire record. It does this + by formatting each record in memory as DKVP (or NIDX, if -a is supplied), using + command-line-specified ORS/OFS/OPS, and matching the resulting line against the + regex specified here. In particular, the regex is not applied to the input + stream: if you have CSV with header line "x,y,z" and data line "1,2,3" then the + regex will be matched, not against either of these lines, but against the DKVP + line "x=1,y=2,z=3". Furthermore, not all the options to system grep are + supported, and this command is intended to be merely a keystroke-saver. To get + all the features of system grep, you can do "mlr --odkvp ... | grep ... | mlr --idkvp ..." 1mgroup-by0m @@ -3359,5 +3358,5 @@ MILLER(1) MILLER(1) - 2023-05-13 MILLER(1) + 2023-06-03 MILLER(1) diff --git a/docs/src/manpage.txt b/docs/src/manpage.txt index 37d960e3d..3d89f78db 100644 --- a/docs/src/manpage.txt +++ b/docs/src/manpage.txt @@ -120,6 +120,7 @@ MILLER(1) MILLER(1) mlr help file-formats Flags: mlr help flags + mlr help flag mlr help list-separator-aliases mlr help list-separator-regex-aliases mlr help comments-in-data-flags @@ -152,6 +153,7 @@ MILLER(1) MILLER(1) mlr help keyword Other: mlr help auxents + mlr help terminals mlr help mlrrc mlr help output-colorization mlr help type-arithmetic-info @@ -780,16 +782,12 @@ MILLER(1) MILLER(1) --rs {string} Specify RS for input and output. 1mAUXILIARY COMMANDS0m - Available subcommands: - aux-list - hex - lecat - termcvt - unhex - help - regtest - repl - version + Available entries: + mlr aux-list + mlr hex + mlr lecat + mlr termcvt + mlr unhex For more information, please invoke mlr {subcommand} --help. 1mMLRRC0m @@ -1182,17 +1180,18 @@ MILLER(1) MILLER(1) Options: -i Use case-insensitive search. -v Invert: pass through records which do not match the regex. + -a Only grep for values, not keys and values. -h|--help Show this message. Note that "mlr filter" is more powerful, but requires you to know field names. - By contrast, "mlr grep" allows you to regex-match the entire record. It does - this by formatting each record in memory as DKVP, using command-line-specified - ORS/OFS/OPS, and matching the resulting line against the regex specified - here. In particular, the regex is not applied to the input stream: if you - have CSV with header line "x,y,z" and data line "1,2,3" then the regex will - be matched, not against either of these lines, but against the DKVP line - "x=1,y=2,z=3". Furthermore, not all the options to system grep are supported, - and this command is intended to be merely a keystroke-saver. To get all the - features of system grep, you can do + By contrast, "mlr grep" allows you to regex-match the entire record. It does this + by formatting each record in memory as DKVP (or NIDX, if -a is supplied), using + command-line-specified ORS/OFS/OPS, and matching the resulting line against the + regex specified here. In particular, the regex is not applied to the input + stream: if you have CSV with header line "x,y,z" and data line "1,2,3" then the + regex will be matched, not against either of these lines, but against the DKVP + line "x=1,y=2,z=3". Furthermore, not all the options to system grep are + supported, and this command is intended to be merely a keystroke-saver. To get + all the features of system grep, you can do "mlr --odkvp ... | grep ... | mlr --idkvp ..." 1mgroup-by0m @@ -3338,4 +3337,4 @@ MILLER(1) MILLER(1) - 2023-05-13 MILLER(1) + 2023-06-03 MILLER(1) diff --git a/docs/src/online-help.md b/docs/src/online-help.md index bdf05fb66..fc6cda292 100644 --- a/docs/src/online-help.md +++ b/docs/src/online-help.md @@ -49,6 +49,7 @@ Essentials: mlr help file-formats Flags: mlr help flags + mlr help flag mlr help list-separator-aliases mlr help list-separator-regex-aliases mlr help comments-in-data-flags @@ -81,6 +82,7 @@ Keywords: mlr help keyword Other: mlr help auxents + mlr help terminals mlr help mlrrc mlr help output-colorization mlr help type-arithmetic-info diff --git a/docs/src/reference-main-auxiliary-commands.md b/docs/src/reference-main-auxiliary-commands.md index a55ce0361..16fa67c09 100644 --- a/docs/src/reference-main-auxiliary-commands.md +++ b/docs/src/reference-main-auxiliary-commands.md @@ -22,16 +22,12 @@ There are a few nearly-standalone programs which have a little to do with the re mlr aux-list
-Available subcommands:
- aux-list
- hex
- lecat
- termcvt
- unhex
- help
- regtest
- repl
- version
+Available entries:
+ mlr aux-list
+ mlr hex
+ mlr lecat
+ mlr termcvt
+ mlr unhex
For more information, please invoke mlr {subcommand} --help.
diff --git a/docs/src/reference-verbs.md b/docs/src/reference-verbs.md
index b0f1d43f5..0558f99d5 100644
--- a/docs/src/reference-verbs.md
+++ b/docs/src/reference-verbs.md
@@ -1325,17 +1325,18 @@ Passes through records which match the regular expression.
Options:
-i Use case-insensitive search.
-v Invert: pass through records which do not match the regex.
+-a Only grep for values, not keys and values.
-h|--help Show this message.
Note that "mlr filter" is more powerful, but requires you to know field names.
-By contrast, "mlr grep" allows you to regex-match the entire record. It does
-this by formatting each record in memory as DKVP, using command-line-specified
-ORS/OFS/OPS, and matching the resulting line against the regex specified
-here. In particular, the regex is not applied to the input stream: if you
-have CSV with header line "x,y,z" and data line "1,2,3" then the regex will
-be matched, not against either of these lines, but against the DKVP line
-"x=1,y=2,z=3". Furthermore, not all the options to system grep are supported,
-and this command is intended to be merely a keystroke-saver. To get all the
-features of system grep, you can do
+By contrast, "mlr grep" allows you to regex-match the entire record. It does this
+by formatting each record in memory as DKVP (or NIDX, if -a is supplied), using
+command-line-specified ORS/OFS/OPS, and matching the resulting line against the
+regex specified here. In particular, the regex is not applied to the input
+stream: if you have CSV with header line "x,y,z" and data line "1,2,3" then the
+regex will be matched, not against either of these lines, but against the DKVP
+line "x=1,y=2,z=3". Furthermore, not all the options to system grep are
+supported, and this command is intended to be merely a keystroke-saver. To get
+all the features of system grep, you can do
"mlr --odkvp ... | grep ... | mlr --idkvp ..."
diff --git a/internal/pkg/mlrval/mlrmap_print.go b/internal/pkg/mlrval/mlrmap_print.go
index 9cfd8122a..5ab7c2d5a 100644
--- a/internal/pkg/mlrval/mlrmap_print.go
+++ b/internal/pkg/mlrval/mlrmap_print.go
@@ -28,6 +28,17 @@ func (mlrmap *Mlrmap) ToDKVPString() string {
return buffer.String()
}
+func (mlrmap *Mlrmap) ToNIDXString() string {
+ var buffer bytes.Buffer // stdio is non-buffered in Go, so buffer for ~5x speed increase
+ for pe := mlrmap.Head; pe != nil; pe = pe.Next {
+ buffer.WriteString(pe.Value.String())
+ if pe.Next != nil {
+ buffer.WriteString(",")
+ }
+ }
+ return buffer.String()
+}
+
// ----------------------------------------------------------------
// Must have non-pointer receiver in order to implement the fmt.Stringer
// interface to make mlrmap printable via fmt.Println et al.
diff --git a/internal/pkg/transformers/grep.go b/internal/pkg/transformers/grep.go
index c009e7219..6e692364a 100644
--- a/internal/pkg/transformers/grep.go
+++ b/internal/pkg/transformers/grep.go
@@ -30,18 +30,19 @@ func transformerGrepUsage(
fmt.Fprint(o, "Options:\n")
fmt.Fprint(o, "-i Use case-insensitive search.\n")
fmt.Fprint(o, "-v Invert: pass through records which do not match the regex.\n")
+ fmt.Fprint(o, "-a Only grep for values, not keys and values.\n")
fmt.Fprintf(o, "-h|--help Show this message.\n")
fmt.Fprintf(o, `Note that "%s filter" is more powerful, but requires you to know field names.
-By contrast, "%s grep" allows you to regex-match the entire record. It does
-this by formatting each record in memory as DKVP, using command-line-specified
-ORS/OFS/OPS, and matching the resulting line against the regex specified
-here. In particular, the regex is not applied to the input stream: if you
-have CSV with header line "x,y,z" and data line "1,2,3" then the regex will
-be matched, not against either of these lines, but against the DKVP line
-"x=1,y=2,z=3". Furthermore, not all the options to system grep are supported,
-and this command is intended to be merely a keystroke-saver. To get all the
-features of system grep, you can do
+By contrast, "%s grep" allows you to regex-match the entire record. It does this
+by formatting each record in memory as DKVP (or NIDX, if -a is supplied), using
+command-line-specified ORS/OFS/OPS, and matching the resulting line against the
+regex specified here. In particular, the regex is not applied to the input
+stream: if you have CSV with header line "x,y,z" and data line "1,2,3" then the
+regex will be matched, not against either of these lines, but against the DKVP
+line "x=1,y=2,z=3". Furthermore, not all the options to system grep are
+supported, and this command is intended to be merely a keystroke-saver. To get
+all the features of system grep, you can do
"%s --odkvp ... | grep ... | %s --idkvp ..."
`, "mlr", "mlr", "mlr", "mlr")
}
@@ -61,6 +62,7 @@ func transformerGrepParseCLI(
ignoreCase := false
invert := false
+ valuesOnly := false
for argi < argc /* variable increment: 1 or 2 depending on flag */ {
opt := args[argi]
@@ -82,6 +84,9 @@ func transformerGrepParseCLI(
} else if opt == "-v" {
invert = true
+ } else if opt == "-a" {
+ valuesOnly = true
+
} else {
transformerGrepUsage(os.Stderr)
os.Exit(1)
@@ -116,6 +121,7 @@ func transformerGrepParseCLI(
transformer, err := NewTransformerGrep(
regexp,
invert,
+ valuesOnly,
)
if err != nil {
fmt.Fprintln(os.Stderr, err)
@@ -127,17 +133,20 @@ func transformerGrepParseCLI(
// ----------------------------------------------------------------
type TransformerGrep struct {
- regexp *regexp.Regexp
- invert bool
+ regexp *regexp.Regexp
+ invert bool
+ valuesOnly bool
}
func NewTransformerGrep(
regexp *regexp.Regexp,
invert bool,
+ valuesOnly bool,
) (*TransformerGrep, error) {
tr := &TransformerGrep{
- regexp: regexp,
- invert: invert,
+ regexp: regexp,
+ invert: invert,
+ valuesOnly: valuesOnly,
}
return tr, nil
}
@@ -153,7 +162,12 @@ func (tr *TransformerGrep) Transform(
HandleDefaultDownstreamDone(inputDownstreamDoneChannel, outputDownstreamDoneChannel)
if !inrecAndContext.EndOfStream {
inrec := inrecAndContext.Record
- inrecAsString := inrec.ToDKVPString()
+ var inrecAsString string
+ if tr.valuesOnly {
+ inrecAsString = inrec.ToNIDXString()
+ } else {
+ inrecAsString = inrec.ToDKVPString()
+ }
matches := tr.regexp.Match([]byte(inrecAsString))
if tr.invert {
if !matches {
diff --git a/man/manpage.txt b/man/manpage.txt
index 37d960e3d..3d89f78db 100644
--- a/man/manpage.txt
+++ b/man/manpage.txt
@@ -120,6 +120,7 @@ MILLER(1) MILLER(1)
mlr help file-formats
Flags:
mlr help flags
+ mlr help flag
mlr help list-separator-aliases
mlr help list-separator-regex-aliases
mlr help comments-in-data-flags
@@ -152,6 +153,7 @@ MILLER(1) MILLER(1)
mlr help keyword
Other:
mlr help auxents
+ mlr help terminals
mlr help mlrrc
mlr help output-colorization
mlr help type-arithmetic-info
@@ -780,16 +782,12 @@ MILLER(1) MILLER(1)
--rs {string} Specify RS for input and output.
1mAUXILIARY COMMANDS0m
- Available subcommands:
- aux-list
- hex
- lecat
- termcvt
- unhex
- help
- regtest
- repl
- version
+ Available entries:
+ mlr aux-list
+ mlr hex
+ mlr lecat
+ mlr termcvt
+ mlr unhex
For more information, please invoke mlr {subcommand} --help.
1mMLRRC0m
@@ -1182,17 +1180,18 @@ MILLER(1) MILLER(1)
Options:
-i Use case-insensitive search.
-v Invert: pass through records which do not match the regex.
+ -a Only grep for values, not keys and values.
-h|--help Show this message.
Note that "mlr filter" is more powerful, but requires you to know field names.
- By contrast, "mlr grep" allows you to regex-match the entire record. It does
- this by formatting each record in memory as DKVP, using command-line-specified
- ORS/OFS/OPS, and matching the resulting line against the regex specified
- here. In particular, the regex is not applied to the input stream: if you
- have CSV with header line "x,y,z" and data line "1,2,3" then the regex will
- be matched, not against either of these lines, but against the DKVP line
- "x=1,y=2,z=3". Furthermore, not all the options to system grep are supported,
- and this command is intended to be merely a keystroke-saver. To get all the
- features of system grep, you can do
+ By contrast, "mlr grep" allows you to regex-match the entire record. It does this
+ by formatting each record in memory as DKVP (or NIDX, if -a is supplied), using
+ command-line-specified ORS/OFS/OPS, and matching the resulting line against the
+ regex specified here. In particular, the regex is not applied to the input
+ stream: if you have CSV with header line "x,y,z" and data line "1,2,3" then the
+ regex will be matched, not against either of these lines, but against the DKVP
+ line "x=1,y=2,z=3". Furthermore, not all the options to system grep are
+ supported, and this command is intended to be merely a keystroke-saver. To get
+ all the features of system grep, you can do
"mlr --odkvp ... | grep ... | mlr --idkvp ..."
1mgroup-by0m
@@ -3338,4 +3337,4 @@ MILLER(1) MILLER(1)
- 2023-05-13 MILLER(1)
+ 2023-06-03 MILLER(1)
diff --git a/man/mlr.1 b/man/mlr.1
index 00a67f9ec..08ad3e578 100644
--- a/man/mlr.1
+++ b/man/mlr.1
@@ -2,12 +2,12 @@
.\" Title: mlr
.\" Author: [see the "AUTHOR" section]
.\" Generator: ./mkman.rb
-.\" Date: 2023-05-13
+.\" Date: 2023-06-03
.\" Manual: \ \&
.\" Source: \ \&
.\" Language: English
.\"
-.TH "MILLER" "1" "2023-05-13" "\ \&" "\ \&"
+.TH "MILLER" "1" "2023-06-03" "\ \&" "\ \&"
.\" -----------------------------------------------------------------
.\" * Portability definitions
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -155,6 +155,7 @@ Essentials:
mlr help file-formats
Flags:
mlr help flags
+ mlr help flag
mlr help list-separator-aliases
mlr help list-separator-regex-aliases
mlr help comments-in-data-flags
@@ -187,6 +188,7 @@ Keywords:
mlr help keyword
Other:
mlr help auxents
+ mlr help terminals
mlr help mlrrc
mlr help output-colorization
mlr help type-arithmetic-info
@@ -937,16 +939,12 @@ Notes about all other separators:
.RS 0
.\}
.nf
-Available subcommands:
- aux-list
- hex
- lecat
- termcvt
- unhex
- help
- regtest
- repl
- version
+Available entries:
+ mlr aux-list
+ mlr hex
+ mlr lecat
+ mlr termcvt
+ mlr unhex
For more information, please invoke mlr {subcommand} --help.
.fi
.if n \{\
@@ -1473,17 +1471,18 @@ Passes through records which match the regular expression.
Options:
-i Use case-insensitive search.
-v Invert: pass through records which do not match the regex.
+-a Only grep for values, not keys and values.
-h|--help Show this message.
Note that "mlr filter" is more powerful, but requires you to know field names.
-By contrast, "mlr grep" allows you to regex-match the entire record. It does
-this by formatting each record in memory as DKVP, using command-line-specified
-ORS/OFS/OPS, and matching the resulting line against the regex specified
-here. In particular, the regex is not applied to the input stream: if you
-have CSV with header line "x,y,z" and data line "1,2,3" then the regex will
-be matched, not against either of these lines, but against the DKVP line
-"x=1,y=2,z=3". Furthermore, not all the options to system grep are supported,
-and this command is intended to be merely a keystroke-saver. To get all the
-features of system grep, you can do
+By contrast, "mlr grep" allows you to regex-match the entire record. It does this
+by formatting each record in memory as DKVP (or NIDX, if -a is supplied), using
+command-line-specified ORS/OFS/OPS, and matching the resulting line against the
+regex specified here. In particular, the regex is not applied to the input
+stream: if you have CSV with header line "x,y,z" and data line "1,2,3" then the
+regex will be matched, not against either of these lines, but against the DKVP
+line "x=1,y=2,z=3". Furthermore, not all the options to system grep are
+supported, and this command is intended to be merely a keystroke-saver. To get
+all the features of system grep, you can do
"mlr --odkvp ... | grep ... | mlr --idkvp ..."
.fi
.if n \{\
diff --git a/test/cases/cli-help/0001/expout b/test/cases/cli-help/0001/expout
index 374275e77..fe29d65db 100644
--- a/test/cases/cli-help/0001/expout
+++ b/test/cases/cli-help/0001/expout
@@ -343,17 +343,18 @@ Passes through records which match the regular expression.
Options:
-i Use case-insensitive search.
-v Invert: pass through records which do not match the regex.
+-a Only grep for values, not keys and values.
-h|--help Show this message.
Note that "mlr filter" is more powerful, but requires you to know field names.
-By contrast, "mlr grep" allows you to regex-match the entire record. It does
-this by formatting each record in memory as DKVP, using command-line-specified
-ORS/OFS/OPS, and matching the resulting line against the regex specified
-here. In particular, the regex is not applied to the input stream: if you
-have CSV with header line "x,y,z" and data line "1,2,3" then the regex will
-be matched, not against either of these lines, but against the DKVP line
-"x=1,y=2,z=3". Furthermore, not all the options to system grep are supported,
-and this command is intended to be merely a keystroke-saver. To get all the
-features of system grep, you can do
+By contrast, "mlr grep" allows you to regex-match the entire record. It does this
+by formatting each record in memory as DKVP (or NIDX, if -a is supplied), using
+command-line-specified ORS/OFS/OPS, and matching the resulting line against the
+regex specified here. In particular, the regex is not applied to the input
+stream: if you have CSV with header line "x,y,z" and data line "1,2,3" then the
+regex will be matched, not against either of these lines, but against the DKVP
+line "x=1,y=2,z=3". Furthermore, not all the options to system grep are
+supported, and this command is intended to be merely a keystroke-saver. To get
+all the features of system grep, you can do
"mlr --odkvp ... | grep ... | mlr --idkvp ..."
================================================================
diff --git a/test/cases/verb-grep/0006/cmd b/test/cases/verb-grep/0006/cmd
new file mode 100644
index 000000000..a6abade76
--- /dev/null
+++ b/test/cases/verb-grep/0006/cmd
@@ -0,0 +1 @@
+mlr --opprint --from test/input/s.dkvp grep -a y
diff --git a/test/cases/verb-grep/0006/experr b/test/cases/verb-grep/0006/experr
new file mode 100644
index 000000000..e69de29bb
diff --git a/test/cases/verb-grep/0006/expout b/test/cases/verb-grep/0006/expout
new file mode 100644
index 000000000..d1c34e562
--- /dev/null
+++ b/test/cases/verb-grep/0006/expout
@@ -0,0 +1,3 @@
+a b i x y
+wye wye 3 0.20460331 0.33831853
+eks wye 4 0.38139939 0.13418874