mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-17 16:38:54 +00:00
* Add named profile sections to .mlrrc, selected via --profile / -P (#358) .mlrrc files may now contain INI-style named sections ("profiles"): # Global settings, applied always: icsv [j] ojson jvstack [tsvout] otsv Lines before any section header are global settings, applied always, so existing .mlrrc files behave exactly as before. The new main flag --profile {name} (alias -P {name}) applies the settings from the [name] section after the global settings; without it, sections are ignored entirely (their lines aren't even parsed). It's a fatal error if a requested profile has no matching section in any .mlrrc file processed, if no .mlrrc file was found at all, or if --profile is combined with --norc or MLRRC=__none__. Also fixes the regression-tester to restore per-case environment variables to their prior values after each case, rather than setting them to the empty string -- needed so cases pointing MLRRC at a test file don't clobber the suite-wide MLRRC=__none__ guard. Includes unit tests, regression-test cases, and doc/man-page updates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Reject --profile / -P within .mlrrc files (#358) Per maintainer feedback on the PR: profiles are selected on the mlr command line, not from within a .mlrrc file, so --profile / -P inside a .mlrrc file is now a parse error -- the same way --prepipe is rejected there -- rather than being silently ignored. Adds a unit test, a should-fail regression case, and a docs bullet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
e415682e61
commit
75275fdcbe
42 changed files with 588 additions and 42 deletions
|
|
@ -64,7 +64,7 @@ one-line summary (here trimmed, and then counted, using Miller itself):
|
|||
<pre class="pre-non-highlight-in-pair">
|
||||
[
|
||||
{
|
||||
"count": 665
|
||||
"count": 666
|
||||
}
|
||||
]
|
||||
</pre>
|
||||
|
|
|
|||
|
|
@ -80,6 +80,57 @@ jlistwrap
|
|||
skip-comments-with @
|
||||
</pre>
|
||||
|
||||
## Named profiles in your .mlrrc
|
||||
|
||||
You can group settings into INI-style named sections, called _profiles_, and select one with the
|
||||
`--profile {name}` main flag, or its alias `-P {name}`. For example, given
|
||||
|
||||
<pre class="pre-non-highlight-non-pair">
|
||||
# Global settings, applied always:
|
||||
icsv
|
||||
|
||||
[j]
|
||||
# Settings applied only with mlr --profile j (or mlr -P j):
|
||||
ojson
|
||||
jvstack
|
||||
|
||||
[tsvout]
|
||||
# Settings applied only with mlr --profile tsvout (or mlr -P tsvout):
|
||||
otsv
|
||||
</pre>
|
||||
|
||||
then `mlr cat myfile.csv` reads CSV and writes DKVP (the global setting applies, and the
|
||||
sections are ignored), while `mlr -P j cat myfile.csv` reads CSV and writes vertically stacked
|
||||
JSON (the global setting applies first, then the settings from the `[j]` section).
|
||||
|
||||
Semantics:
|
||||
|
||||
* Lines before any `[name]` section header are global settings, and are always applied. A
|
||||
`.mlrrc` file without any section headers behaves just as it did in older versions of Miller.
|
||||
|
||||
* With `--profile {name}` (or `-P {name}`), global settings are applied first, then the settings
|
||||
from the `[name]` section. It's a fatal error if no `[name]` section exists in any `.mlrrc`
|
||||
file processed -- or if no `.mlrrc` file was found at all.
|
||||
|
||||
* Without `--profile`, sections are ignored entirely -- their lines aren't even parsed -- so a
|
||||
typo inside an unused profile won't affect your other invocations of `mlr`.
|
||||
|
||||
* Section names are matched exactly (case-sensitively). Whitespace around and within the
|
||||
brackets is ignored: `[ j ]` is the same as `[j]`. Comments are allowed after section headers.
|
||||
|
||||
* If the same section name appears more than once, the settings from all its blocks are applied,
|
||||
in the order they appear in the file.
|
||||
|
||||
* If both `$HOME/.mlrrc` and `./.mlrrc` are processed, each file's global settings and matching
|
||||
section settings are applied in that per-file order, `$HOME/.mlrrc` first. The selected profile
|
||||
needs to exist in only one of them.
|
||||
|
||||
* Since `--profile` selects a section of your `.mlrrc`, it can't be combined with `--norc`, or
|
||||
with `MLRRC=__none__` in the environment -- that's a fatal error.
|
||||
|
||||
* Profiles are selected on the `mlr` command line, not from within a `.mlrrc` file: putting
|
||||
`--profile` (or `-P`) inside a `.mlrrc` file is a parse error, just as `--prepipe` is.
|
||||
|
||||
## Where to put your .mlrrc
|
||||
|
||||
If the environment variable `MLRRC` is set:
|
||||
|
|
|
|||
|
|
@ -48,6 +48,57 @@ Here is an example `.mlrrc` file:
|
|||
|
||||
GENMD-INCLUDE-ESCAPED(sample_mlrrc)
|
||||
|
||||
## Named profiles in your .mlrrc
|
||||
|
||||
You can group settings into INI-style named sections, called _profiles_, and select one with the
|
||||
`--profile {name}` main flag, or its alias `-P {name}`. For example, given
|
||||
|
||||
GENMD-CARDIFY
|
||||
# Global settings, applied always:
|
||||
icsv
|
||||
|
||||
[j]
|
||||
# Settings applied only with mlr --profile j (or mlr -P j):
|
||||
ojson
|
||||
jvstack
|
||||
|
||||
[tsvout]
|
||||
# Settings applied only with mlr --profile tsvout (or mlr -P tsvout):
|
||||
otsv
|
||||
GENMD-EOF
|
||||
|
||||
then `mlr cat myfile.csv` reads CSV and writes DKVP (the global setting applies, and the
|
||||
sections are ignored), while `mlr -P j cat myfile.csv` reads CSV and writes vertically stacked
|
||||
JSON (the global setting applies first, then the settings from the `[j]` section).
|
||||
|
||||
Semantics:
|
||||
|
||||
* Lines before any `[name]` section header are global settings, and are always applied. A
|
||||
`.mlrrc` file without any section headers behaves just as it did in older versions of Miller.
|
||||
|
||||
* With `--profile {name}` (or `-P {name}`), global settings are applied first, then the settings
|
||||
from the `[name]` section. It's a fatal error if no `[name]` section exists in any `.mlrrc`
|
||||
file processed -- or if no `.mlrrc` file was found at all.
|
||||
|
||||
* Without `--profile`, sections are ignored entirely -- their lines aren't even parsed -- so a
|
||||
typo inside an unused profile won't affect your other invocations of `mlr`.
|
||||
|
||||
* Section names are matched exactly (case-sensitively). Whitespace around and within the
|
||||
brackets is ignored: `[ j ]` is the same as `[j]`. Comments are allowed after section headers.
|
||||
|
||||
* If the same section name appears more than once, the settings from all its blocks are applied,
|
||||
in the order they appear in the file.
|
||||
|
||||
* If both `$HOME/.mlrrc` and `./.mlrrc` are processed, each file's global settings and matching
|
||||
section settings are applied in that per-file order, `$HOME/.mlrrc` first. The selected profile
|
||||
needs to exist in only one of them.
|
||||
|
||||
* Since `--profile` selects a section of your `.mlrrc`, it can't be combined with `--norc`, or
|
||||
with `MLRRC=__none__` in the environment -- that's a fatal error.
|
||||
|
||||
* Profiles are selected on the `mlr` command line, not from within a `.mlrrc` file: putting
|
||||
`--profile` (or `-P`) inside a `.mlrrc` file is a parse error, just as `--prepipe` is.
|
||||
|
||||
## Where to put your .mlrrc
|
||||
|
||||
If the environment variable `MLRRC` is set:
|
||||
|
|
|
|||
|
|
@ -673,6 +673,11 @@ This is simply a copy of what you should see on running `man mlr` at a command p
|
|||
--ofmte {n} Use --ofmte 6 as shorthand for --ofmt %.6e, etc.
|
||||
--ofmtf {n} Use --ofmtf 6 as shorthand for --ofmt %.6f, etc.
|
||||
--ofmtg {n} Use --ofmtg 6 as shorthand for --ofmt %.6g, etc.
|
||||
--profile or -P {name} Apply the settings from the [name] section of your
|
||||
.mlrrc file, after any global (pre-section) settings.
|
||||
It's an error if no such section exists in any .mlrrc
|
||||
file processed. For more information please see
|
||||
https://miller.readthedocs.io/en/latest/customization/.
|
||||
--records-per-batch {n} This is an internal parameter for maximum number of
|
||||
records in a batch size. Normally this does not need
|
||||
to be modified, except when input is from `tail -f`.
|
||||
|
|
|
|||
|
|
@ -652,6 +652,11 @@
|
|||
--ofmte {n} Use --ofmte 6 as shorthand for --ofmt %.6e, etc.
|
||||
--ofmtf {n} Use --ofmtf 6 as shorthand for --ofmt %.6f, etc.
|
||||
--ofmtg {n} Use --ofmtg 6 as shorthand for --ofmt %.6g, etc.
|
||||
--profile or -P {name} Apply the settings from the [name] section of your
|
||||
.mlrrc file, after any global (pre-section) settings.
|
||||
It's an error if no such section exists in any .mlrrc
|
||||
file processed. For more information please see
|
||||
https://miller.readthedocs.io/en/latest/customization/.
|
||||
--records-per-batch {n} This is an internal parameter for maximum number of
|
||||
records in a batch size. Normally this does not need
|
||||
to be modified, except when input is from `tail -f`.
|
||||
|
|
|
|||
|
|
@ -320,6 +320,7 @@ These are flags which don't fit into any other category.
|
|||
* `--ofmte {n}`: Use --ofmte 6 as shorthand for --ofmt %.6e, etc.
|
||||
* `--ofmtf {n}`: Use --ofmtf 6 as shorthand for --ofmt %.6f, etc.
|
||||
* `--ofmtg {n}`: Use --ofmtg 6 as shorthand for --ofmt %.6g, etc.
|
||||
* `--profile or -P {name}`: Apply the settings from the [name] section of your .mlrrc file, after any global (pre-section) settings. It's an error if no such section exists in any .mlrrc file processed. For more information please see https://miller.readthedocs.io/en/latest/customization/.
|
||||
* `--records-per-batch {n}`: This is an internal parameter for maximum number of records in a batch size. Normally this does not need to be modified, except when input is from `tail -f`. See also https://miller.readthedocs.io/en/latest/reference-main-flag-list/.
|
||||
* `--s-no-comment-strip {file name}`: Take command-line flags from file name, like -s, but with no comment-stripping. For more information please see https://miller.readthedocs.io/en/latest/scripting/.
|
||||
* `--seed {n}`: with `n` of the form `12345678` or `0xcafefeed`. For `put`/`filter` `urand`, `urandint`, and `urand32`.
|
||||
|
|
|
|||
|
|
@ -652,6 +652,11 @@
|
|||
--ofmte {n} Use --ofmte 6 as shorthand for --ofmt %.6e, etc.
|
||||
--ofmtf {n} Use --ofmtf 6 as shorthand for --ofmt %.6f, etc.
|
||||
--ofmtg {n} Use --ofmtg 6 as shorthand for --ofmt %.6g, etc.
|
||||
--profile or -P {name} Apply the settings from the [name] section of your
|
||||
.mlrrc file, after any global (pre-section) settings.
|
||||
It's an error if no such section exists in any .mlrrc
|
||||
file processed. For more information please see
|
||||
https://miller.readthedocs.io/en/latest/customization/.
|
||||
--records-per-batch {n} This is an internal parameter for maximum number of
|
||||
records in a batch size. Normally this does not need
|
||||
to be modified, except when input is from `tail -f`.
|
||||
|
|
|
|||
|
|
@ -788,6 +788,11 @@ These are flags which don't fit into any other category.
|
|||
--ofmte {n} Use --ofmte 6 as shorthand for --ofmt %.6e, etc.
|
||||
--ofmtf {n} Use --ofmtf 6 as shorthand for --ofmt %.6f, etc.
|
||||
--ofmtg {n} Use --ofmtg 6 as shorthand for --ofmt %.6g, etc.
|
||||
--profile or -P {name} Apply the settings from the [name] section of your
|
||||
.mlrrc file, after any global (pre-section) settings.
|
||||
It's an error if no such section exists in any .mlrrc
|
||||
file processed. For more information please see
|
||||
https://miller.readthedocs.io/en/latest/customization/.
|
||||
--records-per-batch {n} This is an internal parameter for maximum number of
|
||||
records in a batch size. Normally this does not need
|
||||
to be modified, except when input is from `tail -f`.
|
||||
|
|
|
|||
|
|
@ -3789,5 +3789,23 @@ has its own overhead.`,
|
|||
*pargi += 1
|
||||
},
|
||||
},
|
||||
|
||||
{
|
||||
name: "--profile",
|
||||
altNames: []string{"-P"},
|
||||
arg: "{name}",
|
||||
help: `Apply the settings from the [name] section of your .mlrrc file, after any global
|
||||
(pre-section) settings. It's an error if no such section exists in any .mlrrc file processed.
|
||||
For more information please see ` + lib.DOC_URL + `/en/latest/customization/.`,
|
||||
parser: func(args []string, argc int, pargi *int, options *TOptions) {
|
||||
CheckArgCount(args, *pargi, argc, 2)
|
||||
// The actual profile selection is handled in pkg/climain,
|
||||
// before the .mlrrc file is loaded, which is in turn before
|
||||
// main-flag sequences are parsed into the options struct.
|
||||
// Registration here ensures the flag is recognized (not an
|
||||
// error) during flag parsing, and appears in --help.
|
||||
*pargi += 2
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,49 +11,127 @@ import (
|
|||
"github.com/johnkerl/miller/v6/pkg/cli"
|
||||
)
|
||||
|
||||
// loadMlrrcOrDie rule: If $MLRRC is set, use it and only it. Otherwise try
|
||||
// first $HOME/.mlrrc and then ./.mlrrc but let them stack: e.g. $HOME/.mlrrc
|
||||
// is lots of settings and maybe in one subdir you want to override just a
|
||||
// setting or two.
|
||||
// loadMlrrcOrDie is a fatal-error wrapper around loadMlrrc.
|
||||
func loadMlrrcOrDie(
|
||||
options *cli.TOptions,
|
||||
profileName string,
|
||||
) {
|
||||
err := loadMlrrc(options, profileName)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "mlr: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// loadMlrrc rule: If $MLRRC is set, use it and only it. Otherwise try first
|
||||
// $HOME/.mlrrc and then ./.mlrrc but let them stack: e.g. $HOME/.mlrrc is
|
||||
// lots of settings and maybe in one subdir you want to override just a
|
||||
// setting or two.
|
||||
//
|
||||
// The profileName argument comes from the --profile {name} / -P {name} main
|
||||
// flag. Empty string means no profile was requested: only global (pre-section)
|
||||
// lines of the .mlrrc file(s) are applied, and any [section] blocks are
|
||||
// skipped. Non-empty means global lines are applied first, then the lines in
|
||||
// any [profileName] section. It's a fatal error if a profile was requested
|
||||
// but no matching section exists in any .mlrrc file processed.
|
||||
func loadMlrrc(
|
||||
options *cli.TOptions,
|
||||
profileName string,
|
||||
) error {
|
||||
foundProfile := false
|
||||
loadedPaths := []string{}
|
||||
|
||||
env_mlrrc := os.Getenv("MLRRC")
|
||||
|
||||
if env_mlrrc != "" {
|
||||
if env_mlrrc == "__none__" {
|
||||
return
|
||||
if profileName != "" {
|
||||
return fmt.Errorf(
|
||||
"--profile \"%s\" was specified, but .mlrrc processing is disabled since the MLRRC environment variable is set to \"__none__\"",
|
||||
profileName,
|
||||
)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if tryLoadMlrrc(options, env_mlrrc) {
|
||||
return
|
||||
loaded, err := tryLoadMlrrc(options, env_mlrrc, profileName, &foundProfile)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if loaded {
|
||||
return checkMlrrcProfileWasFound(profileName, foundProfile, []string{env_mlrrc})
|
||||
}
|
||||
}
|
||||
|
||||
env_home := os.Getenv("HOME")
|
||||
if env_home != "" {
|
||||
path := env_home + "/.mlrrc"
|
||||
tryLoadMlrrc(options, path)
|
||||
loaded, err := tryLoadMlrrc(options, path, profileName, &foundProfile)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if loaded {
|
||||
loadedPaths = append(loadedPaths, path)
|
||||
}
|
||||
}
|
||||
|
||||
tryLoadMlrrc(options, "./.mlrrc")
|
||||
loaded, err := tryLoadMlrrc(options, "./.mlrrc", profileName, &foundProfile)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if loaded {
|
||||
loadedPaths = append(loadedPaths, "./.mlrrc")
|
||||
}
|
||||
|
||||
return checkMlrrcProfileWasFound(profileName, foundProfile, loadedPaths)
|
||||
}
|
||||
|
||||
// tryLoadMlrrc is a helper function for loadMlrrcOrDie.
|
||||
// checkMlrrcProfileWasFound is a helper function for loadMlrrc: if a profile
|
||||
// was requested via --profile {name} / -P {name}, there must be a matching
|
||||
// [name] section in at least one processed .mlrrc file.
|
||||
func checkMlrrcProfileWasFound(
|
||||
profileName string,
|
||||
foundProfile bool,
|
||||
loadedPaths []string,
|
||||
) error {
|
||||
if profileName == "" || foundProfile {
|
||||
return nil
|
||||
}
|
||||
if len(loadedPaths) == 0 {
|
||||
return fmt.Errorf(
|
||||
"--profile \"%s\" was specified, but no .mlrrc file was found",
|
||||
profileName,
|
||||
)
|
||||
}
|
||||
return fmt.Errorf(
|
||||
"--profile \"%s\" was specified, but no [%s] section was found in %s",
|
||||
profileName, profileName, strings.Join(loadedPaths, ", "),
|
||||
)
|
||||
}
|
||||
|
||||
// tryLoadMlrrc is a helper function for loadMlrrc. The first return value is
|
||||
// whether the file could be opened at all: an unopenable file is not an error
|
||||
// (that's the normal case when no .mlrrc file exists). The second is any
|
||||
// parse error within an opened file.
|
||||
func tryLoadMlrrc(
|
||||
options *cli.TOptions,
|
||||
path string,
|
||||
) bool {
|
||||
profileName string,
|
||||
pFoundProfile *bool,
|
||||
) (bool, error) {
|
||||
handle, err := os.Open(path)
|
||||
if err != nil {
|
||||
return false
|
||||
return false, nil
|
||||
}
|
||||
defer func() { _ = handle.Close() }()
|
||||
|
||||
lineReader := bufio.NewReader(handle)
|
||||
|
||||
eof := false
|
||||
// Empty string means we're before any [section] header: the global part
|
||||
// of the file which is applied unconditionally.
|
||||
currentSectionName := ""
|
||||
|
||||
lineno := 0
|
||||
for !eof {
|
||||
for {
|
||||
line, err := lineReader.ReadString('\n')
|
||||
if err == io.EOF {
|
||||
break
|
||||
|
|
@ -61,43 +139,82 @@ func tryLoadMlrrc(
|
|||
lineno++
|
||||
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "mlr: %v\n", err)
|
||||
os.Exit(1)
|
||||
return false
|
||||
return true, err
|
||||
}
|
||||
|
||||
// This is how to do a chomp:
|
||||
// TODO: handle \r\n with libified solution.
|
||||
line = strings.TrimRight(line, "\n")
|
||||
|
||||
if !handleMlrrcLine(options, line) {
|
||||
fmt.Fprintf(os.Stderr, "%s: parse error at file \"%s\" line %d: %s\n",
|
||||
"mlr", path, lineno, line,
|
||||
// Comment-strip, then left-trim / right-trim.
|
||||
stripped := stripMlrrcLine(line)
|
||||
|
||||
if stripped == "" { // line was whitespace-only, or comment-only
|
||||
continue
|
||||
}
|
||||
|
||||
if strings.HasPrefix(stripped, "[") {
|
||||
sectionName, ok := parseMlrrcSectionHeader(stripped)
|
||||
if !ok {
|
||||
return true, fmt.Errorf(
|
||||
"parse error at file \"%s\" line %d: %s", path, lineno, line,
|
||||
)
|
||||
}
|
||||
currentSectionName = sectionName
|
||||
if profileName != "" && sectionName == profileName {
|
||||
*pFoundProfile = true
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
// Global (pre-section) lines are always applied. Section lines are
|
||||
// applied only if their section is the requested profile; lines in
|
||||
// other sections are skipped entirely (not even parsed), so that a
|
||||
// typo within an unused profile doesn't break every mlr invocation.
|
||||
if currentSectionName != "" && currentSectionName != profileName {
|
||||
continue
|
||||
}
|
||||
|
||||
if !handleMlrrcLine(options, stripped) {
|
||||
return true, fmt.Errorf(
|
||||
"parse error at file \"%s\" line %d: %s", path, lineno, line,
|
||||
)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
return true
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// handleMlrrcLine is a helper function for loadMlrrcOrDie.
|
||||
// stripMlrrcLine removes any comment (from '#' to end of line) and
|
||||
// surrounding whitespace from a .mlrrc line.
|
||||
func stripMlrrcLine(line string) string {
|
||||
re := regexp.MustCompile("#.*")
|
||||
line = re.ReplaceAllString(line, "")
|
||||
return strings.TrimSpace(line)
|
||||
}
|
||||
|
||||
// parseMlrrcSectionHeader parses an INI-style section header like "[name]".
|
||||
// The input must already be comment-stripped and whitespace-trimmed, and
|
||||
// start with '['. Whitespace within the brackets is allowed: "[ name ]" is
|
||||
// the same as "[name]". Returns the section name, and false if the line is
|
||||
// not a well-formed section header.
|
||||
func parseMlrrcSectionHeader(line string) (string, bool) {
|
||||
if !strings.HasSuffix(line, "]") {
|
||||
return "", false
|
||||
}
|
||||
sectionName := strings.TrimSpace(line[1 : len(line)-1])
|
||||
if sectionName == "" || strings.ContainsAny(sectionName, "[]") {
|
||||
return "", false
|
||||
}
|
||||
return sectionName, true
|
||||
}
|
||||
|
||||
// handleMlrrcLine is a helper function for tryLoadMlrrc, handling a single
|
||||
// (comment-stripped, whitespace-trimmed, non-empty) settings line.
|
||||
func handleMlrrcLine(
|
||||
options *cli.TOptions,
|
||||
line string,
|
||||
) bool {
|
||||
|
||||
// Comment-strip
|
||||
re := regexp.MustCompile("#.*")
|
||||
line = re.ReplaceAllString(line, "")
|
||||
|
||||
// Left-trim / right-trim
|
||||
line = strings.TrimSpace(line)
|
||||
|
||||
if line == "" { // line was whitespace-only
|
||||
return true
|
||||
}
|
||||
|
||||
// Prepend initial "--" if it's not already there
|
||||
if !strings.HasPrefix(line, "-") {
|
||||
line = "--" + line
|
||||
|
|
@ -114,6 +231,10 @@ func handleMlrrcLine(
|
|||
} else if args[0] == "--load" || args[0] == "--mload" {
|
||||
// Don't allow code execution via .mlrrc
|
||||
return false
|
||||
} else if args[0] == "--profile" || args[0] == "-P" {
|
||||
// Profiles are selected on the mlr command line, not from within a
|
||||
// .mlrrc file
|
||||
return false
|
||||
} else if cli.FLAG_TABLE.Parse(args, argc, &argi, options) {
|
||||
// handled
|
||||
} else {
|
||||
|
|
|
|||
219
pkg/climain/mlrcli_mlrrc_test.go
Normal file
219
pkg/climain/mlrcli_mlrrc_test.go
Normal file
|
|
@ -0,0 +1,219 @@
|
|||
package climain
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/johnkerl/miller/v6/pkg/cli"
|
||||
)
|
||||
|
||||
// writeTempMlrrc writes contents to a file in a temp directory and points the
|
||||
// MLRRC environment variable at it, so loadMlrrc reads it and only it.
|
||||
func writeTempMlrrc(t *testing.T, contents string) {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "mlrrc")
|
||||
if err := os.WriteFile(path, []byte(contents), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("MLRRC", path)
|
||||
}
|
||||
|
||||
const testMlrrcContents = `# A global setting, applied always:
|
||||
icsv
|
||||
|
||||
[j]
|
||||
ojson # A comment after a setting
|
||||
jvstack
|
||||
|
||||
[p]
|
||||
opprint
|
||||
|
||||
[j]
|
||||
no-auto-flatten
|
||||
`
|
||||
|
||||
func TestMlrrcGlobalOnlyBackCompat(t *testing.T) {
|
||||
// No sections at all: everything applies -- this is the historical
|
||||
// .mlrrc format.
|
||||
writeTempMlrrc(t, "icsv\nojson\n")
|
||||
options := cli.DefaultOptions()
|
||||
if err := loadMlrrc(options, ""); err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if options.ReaderOptions.InputFileFormat != "csv" {
|
||||
t.Errorf("input format: got %s, want csv", options.ReaderOptions.InputFileFormat)
|
||||
}
|
||||
if options.WriterOptions.OutputFileFormat != "json" {
|
||||
t.Errorf("output format: got %s, want json", options.WriterOptions.OutputFileFormat)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMlrrcSectionsIgnoredWithoutProfile(t *testing.T) {
|
||||
writeTempMlrrc(t, testMlrrcContents)
|
||||
options := cli.DefaultOptions()
|
||||
if err := loadMlrrc(options, ""); err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if options.ReaderOptions.InputFileFormat != "csv" {
|
||||
t.Errorf("input format: got %s, want csv", options.ReaderOptions.InputFileFormat)
|
||||
}
|
||||
// The [j] and [p] sections must not be applied.
|
||||
if options.WriterOptions.OutputFileFormat != "dkvp" {
|
||||
t.Errorf("output format: got %s, want dkvp", options.WriterOptions.OutputFileFormat)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMlrrcProfileSelection(t *testing.T) {
|
||||
writeTempMlrrc(t, testMlrrcContents)
|
||||
options := cli.DefaultOptions()
|
||||
if err := loadMlrrc(options, "j"); err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
// Global setting applies first ...
|
||||
if options.ReaderOptions.InputFileFormat != "csv" {
|
||||
t.Errorf("input format: got %s, want csv", options.ReaderOptions.InputFileFormat)
|
||||
}
|
||||
// ... then the [j] settings.
|
||||
if options.WriterOptions.OutputFileFormat != "json" {
|
||||
t.Errorf("output format: got %s, want json", options.WriterOptions.OutputFileFormat)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMlrrcRepeatedSectionsAccumulate(t *testing.T) {
|
||||
writeTempMlrrc(t, testMlrrcContents)
|
||||
options := cli.DefaultOptions()
|
||||
if err := loadMlrrc(options, "j"); err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
// no-auto-flatten is in the second [j] block.
|
||||
if options.WriterOptions.AutoFlatten {
|
||||
t.Errorf("auto-flatten: got true, want false via second [j] block")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMlrrcOtherProfileNotApplied(t *testing.T) {
|
||||
writeTempMlrrc(t, testMlrrcContents)
|
||||
options := cli.DefaultOptions()
|
||||
if err := loadMlrrc(options, "p"); err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if options.WriterOptions.OutputFileFormat != "pprint" {
|
||||
t.Errorf("output format: got %s, want pprint", options.WriterOptions.OutputFileFormat)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMlrrcMissingProfileIsError(t *testing.T) {
|
||||
writeTempMlrrc(t, testMlrrcContents)
|
||||
options := cli.DefaultOptions()
|
||||
err := loadMlrrc(options, "nonesuch")
|
||||
if err == nil {
|
||||
t.Fatal("expected error for missing profile, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "nonesuch") {
|
||||
t.Errorf("error should name the missing profile: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMlrrcProfileWithMlrrcNoneIsError(t *testing.T) {
|
||||
t.Setenv("MLRRC", "__none__")
|
||||
options := cli.DefaultOptions()
|
||||
err := loadMlrrc(options, "j")
|
||||
if err == nil {
|
||||
t.Fatal("expected error for profile with MLRRC=__none__, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMlrrcProfileWithNoMlrrcFileIsError(t *testing.T) {
|
||||
// Point MLRRC at a nonexistent file: an unopenable file is silently
|
||||
// skipped (the normal no-.mlrrc case), and HOME is remapped to an empty
|
||||
// temp directory, so no .mlrrc file is processed at all.
|
||||
t.Setenv("MLRRC", filepath.Join(t.TempDir(), "nonexistent"))
|
||||
t.Setenv("HOME", t.TempDir())
|
||||
options := cli.DefaultOptions()
|
||||
err := loadMlrrc(options, "j")
|
||||
if err == nil {
|
||||
t.Fatal("expected error for profile with no .mlrrc file, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "no .mlrrc file was found") {
|
||||
t.Errorf("unexpected error message: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMlrrcWhitespaceAndCommentsAroundSectionHeaders(t *testing.T) {
|
||||
writeTempMlrrc(t, "icsv\n\n [ j ] # a comment\nojson\n")
|
||||
options := cli.DefaultOptions()
|
||||
if err := loadMlrrc(options, "j"); err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if options.WriterOptions.OutputFileFormat != "json" {
|
||||
t.Errorf("output format: got %s, want json", options.WriterOptions.OutputFileFormat)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMlrrcMalformedSectionHeadersAreErrors(t *testing.T) {
|
||||
for _, header := range []string{"[j", "[]", "[ ]", "[a[b]"} {
|
||||
writeTempMlrrc(t, header+"\n")
|
||||
options := cli.DefaultOptions()
|
||||
err := loadMlrrc(options, "")
|
||||
if err == nil {
|
||||
t.Errorf("expected parse error for header %q, got nil", header)
|
||||
} else if !strings.Contains(err.Error(), "parse error") {
|
||||
t.Errorf("expected parse error for header %q, got: %v", header, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMlrrcUnusedProfileLinesNotValidated(t *testing.T) {
|
||||
// A typo inside a section which isn't selected must not break other
|
||||
// invocations of mlr.
|
||||
writeTempMlrrc(t, "icsv\n[j]\nthis-is-not-a-flag\n")
|
||||
options := cli.DefaultOptions()
|
||||
if err := loadMlrrc(options, ""); err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
// But it is a parse error when the section is selected.
|
||||
options = cli.DefaultOptions()
|
||||
if err := loadMlrrc(options, "j"); err == nil {
|
||||
t.Fatal("expected parse error for selected profile with bad line, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMlrrcGlobalParseErrorsStillFatal(t *testing.T) {
|
||||
writeTempMlrrc(t, "this-is-not-a-flag\n")
|
||||
options := cli.DefaultOptions()
|
||||
if err := loadMlrrc(options, ""); err == nil {
|
||||
t.Fatal("expected parse error, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMlrrcPrepipeStillDisallowed(t *testing.T) {
|
||||
// Code-execution flags are disallowed in .mlrrc -- inside profiles too.
|
||||
writeTempMlrrc(t, "[j]\nprepipe zcat\n")
|
||||
options := cli.DefaultOptions()
|
||||
if err := loadMlrrc(options, "j"); err == nil {
|
||||
t.Fatal("expected error for --prepipe within a profile, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMlrrcProfileFlagDisallowedWithinMlrrc(t *testing.T) {
|
||||
// Profiles are selected on the mlr command line, not from within a
|
||||
// .mlrrc file: --profile / -P there is a parse error, like --prepipe.
|
||||
for _, line := range []string{"profile j", "--profile j", "-P j"} {
|
||||
writeTempMlrrc(t, line+"\n")
|
||||
options := cli.DefaultOptions()
|
||||
err := loadMlrrc(options, "")
|
||||
if err == nil {
|
||||
t.Errorf("expected parse error for %q within .mlrrc, got nil", line)
|
||||
} else if !strings.Contains(err.Error(), "parse error") {
|
||||
t.Errorf("expected parse error for %q within .mlrrc, got: %v", line, err)
|
||||
}
|
||||
}
|
||||
// Inside a selected profile section, too.
|
||||
writeTempMlrrc(t, "[j]\nprofile p\n")
|
||||
options := cli.DefaultOptions()
|
||||
if err := loadMlrrc(options, "j"); err == nil {
|
||||
t.Fatal("expected parse error for --profile within a profile section, got nil")
|
||||
}
|
||||
}
|
||||
|
|
@ -309,16 +309,30 @@ func parseCommandLinePassTwo(
|
|||
ignoresInput := false
|
||||
|
||||
// Load a .mlrrc file unless --norc was a main-flag on the command line.
|
||||
// If --profile {name} / -P {name} was a main-flag, apply the settings
|
||||
// from that [name] section of the .mlrrc file, after any global
|
||||
// (pre-section) settings. These must be found before the .mlrrc file is
|
||||
// loaded, which is before the main flag-sequences are processed.
|
||||
loadMlrrc := true
|
||||
mlrrcProfileName := ""
|
||||
for _, flagSequence := range flagSequences {
|
||||
lib.InternalCodingErrorIf(len(flagSequence) < 1)
|
||||
if flagSequence[0] == "--norc" {
|
||||
switch flagSequence[0] {
|
||||
case "--norc":
|
||||
loadMlrrc = false
|
||||
break
|
||||
case "--profile", "-P":
|
||||
lib.InternalCodingErrorIf(len(flagSequence) < 2)
|
||||
mlrrcProfileName = flagSequence[1]
|
||||
}
|
||||
}
|
||||
if loadMlrrc {
|
||||
loadMlrrcOrDie(options)
|
||||
loadMlrrcOrDie(options, mlrrcProfileName)
|
||||
} else if mlrrcProfileName != "" {
|
||||
fmt.Fprintf(os.Stderr,
|
||||
"mlr: --profile \"%s\" was specified along with --norc, which disables .mlrrc processing.\n",
|
||||
mlrrcProfileName,
|
||||
)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Process the flag-sequences in order from pass one. We assume all the
|
||||
|
|
|
|||
|
|
@ -454,13 +454,18 @@ func (rt *RegTester) executeSingleCmdFile(
|
|||
|
||||
passed := true
|
||||
|
||||
// Set any case-specific environment variables before running the case.
|
||||
// Set any case-specific environment variables before running the case,
|
||||
// remembering their prior values so they can be restored afterward. E.g.
|
||||
// the suite-wide MLRRC=__none__ must be restored after a case which
|
||||
// points MLRRC at a test file.
|
||||
previousEnvValues := make(map[string]string)
|
||||
for pe := envKeyValuePairs.Head; pe != nil; pe = pe.Next {
|
||||
key := pe.Key
|
||||
value := pe.Value
|
||||
if verbosityLevel >= 3 {
|
||||
fmt.Printf("SETENV %s=%s\n", key, value)
|
||||
}
|
||||
previousEnvValues[key] = os.Getenv(key)
|
||||
_ = os.Setenv(key, value)
|
||||
}
|
||||
// This is so 'mlr' files can find the case-directory if they need it --
|
||||
|
|
@ -488,7 +493,7 @@ func (rt *RegTester) executeSingleCmdFile(
|
|||
actualStdout, actualStderr, actualExitCode := RunMillerCommand(rt.exeName, cmd)
|
||||
// ****************************************************************
|
||||
|
||||
// Unset any case-specific environment variables after running the case.
|
||||
// Restore any case-specific environment variables after running the case.
|
||||
// This is important since the setenv is done in the current process,
|
||||
// and we don't want to affect subsequent test cases.
|
||||
for pe := envKeyValuePairs.Head; pe != nil; pe = pe.Next {
|
||||
|
|
@ -496,7 +501,7 @@ func (rt *RegTester) executeSingleCmdFile(
|
|||
if verbosityLevel >= 3 {
|
||||
fmt.Printf("UNSETENV %s\n", key)
|
||||
}
|
||||
_ = os.Setenv(key, "")
|
||||
_ = os.Setenv(key, previousEnvValues[key])
|
||||
}
|
||||
_ = os.Setenv("CASEDIR", "")
|
||||
|
||||
|
|
|
|||
1
test/cases/cli-mlrrc-profiles/0001/cmd
Normal file
1
test/cases/cli-mlrrc-profiles/0001/cmd
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr cat test/input/a.csv
|
||||
1
test/cases/cli-mlrrc-profiles/0001/env
Normal file
1
test/cases/cli-mlrrc-profiles/0001/env
Normal file
|
|
@ -0,0 +1 @@
|
|||
MLRRC=test/input/mlrrc-with-profiles
|
||||
0
test/cases/cli-mlrrc-profiles/0001/experr
Normal file
0
test/cases/cli-mlrrc-profiles/0001/experr
Normal file
2
test/cases/cli-mlrrc-profiles/0001/expout
Normal file
2
test/cases/cli-mlrrc-profiles/0001/expout
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
a=1,b=2,c=3
|
||||
a=4,b=5,c=6
|
||||
1
test/cases/cli-mlrrc-profiles/0002/cmd
Normal file
1
test/cases/cli-mlrrc-profiles/0002/cmd
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr --profile j cat test/input/a.csv
|
||||
1
test/cases/cli-mlrrc-profiles/0002/env
Normal file
1
test/cases/cli-mlrrc-profiles/0002/env
Normal file
|
|
@ -0,0 +1 @@
|
|||
MLRRC=test/input/mlrrc-with-profiles
|
||||
0
test/cases/cli-mlrrc-profiles/0002/experr
Normal file
0
test/cases/cli-mlrrc-profiles/0002/experr
Normal file
12
test/cases/cli-mlrrc-profiles/0002/expout
Normal file
12
test/cases/cli-mlrrc-profiles/0002/expout
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
[
|
||||
{
|
||||
"a": 1,
|
||||
"b": 2,
|
||||
"c": 3
|
||||
},
|
||||
{
|
||||
"a": 4,
|
||||
"b": 5,
|
||||
"c": 6
|
||||
}
|
||||
]
|
||||
1
test/cases/cli-mlrrc-profiles/0003/cmd
Normal file
1
test/cases/cli-mlrrc-profiles/0003/cmd
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr -P p cat test/input/a.csv
|
||||
1
test/cases/cli-mlrrc-profiles/0003/env
Normal file
1
test/cases/cli-mlrrc-profiles/0003/env
Normal file
|
|
@ -0,0 +1 @@
|
|||
MLRRC=test/input/mlrrc-with-profiles
|
||||
0
test/cases/cli-mlrrc-profiles/0003/experr
Normal file
0
test/cases/cli-mlrrc-profiles/0003/experr
Normal file
3
test/cases/cli-mlrrc-profiles/0003/expout
Normal file
3
test/cases/cli-mlrrc-profiles/0003/expout
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
a b c
|
||||
1 2 3
|
||||
4 5 6
|
||||
1
test/cases/cli-mlrrc-profiles/0004/cmd
Normal file
1
test/cases/cli-mlrrc-profiles/0004/cmd
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr -P nonesuch cat test/input/a.csv
|
||||
1
test/cases/cli-mlrrc-profiles/0004/env
Normal file
1
test/cases/cli-mlrrc-profiles/0004/env
Normal file
|
|
@ -0,0 +1 @@
|
|||
MLRRC=test/input/mlrrc-with-profiles
|
||||
1
test/cases/cli-mlrrc-profiles/0004/experr
Normal file
1
test/cases/cli-mlrrc-profiles/0004/experr
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr: --profile "nonesuch" was specified, but no [nonesuch] section was found in test/input/mlrrc-with-profiles
|
||||
0
test/cases/cli-mlrrc-profiles/0004/expout
Normal file
0
test/cases/cli-mlrrc-profiles/0004/expout
Normal file
0
test/cases/cli-mlrrc-profiles/0004/should-fail
Normal file
0
test/cases/cli-mlrrc-profiles/0004/should-fail
Normal file
1
test/cases/cli-mlrrc-profiles/0005/cmd
Normal file
1
test/cases/cli-mlrrc-profiles/0005/cmd
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr --norc -P j cat test/input/a.csv
|
||||
1
test/cases/cli-mlrrc-profiles/0005/env
Normal file
1
test/cases/cli-mlrrc-profiles/0005/env
Normal file
|
|
@ -0,0 +1 @@
|
|||
MLRRC=test/input/mlrrc-with-profiles
|
||||
1
test/cases/cli-mlrrc-profiles/0005/experr
Normal file
1
test/cases/cli-mlrrc-profiles/0005/experr
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr: --profile "j" was specified along with --norc, which disables .mlrrc processing.
|
||||
0
test/cases/cli-mlrrc-profiles/0005/expout
Normal file
0
test/cases/cli-mlrrc-profiles/0005/expout
Normal file
0
test/cases/cli-mlrrc-profiles/0005/should-fail
Normal file
0
test/cases/cli-mlrrc-profiles/0005/should-fail
Normal file
1
test/cases/cli-mlrrc-profiles/0006/cmd
Normal file
1
test/cases/cli-mlrrc-profiles/0006/cmd
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr cat test/input/a.csv
|
||||
1
test/cases/cli-mlrrc-profiles/0006/env
Normal file
1
test/cases/cli-mlrrc-profiles/0006/env
Normal file
|
|
@ -0,0 +1 @@
|
|||
MLRRC=test/input/mlrrc-with-profile-flag
|
||||
1
test/cases/cli-mlrrc-profiles/0006/experr
Normal file
1
test/cases/cli-mlrrc-profiles/0006/experr
Normal file
|
|
@ -0,0 +1 @@
|
|||
mlr: parse error at file "test/input/mlrrc-with-profile-flag" line 2: profile j
|
||||
0
test/cases/cli-mlrrc-profiles/0006/expout
Normal file
0
test/cases/cli-mlrrc-profiles/0006/expout
Normal file
0
test/cases/cli-mlrrc-profiles/0006/should-fail
Normal file
0
test/cases/cli-mlrrc-profiles/0006/should-fail
Normal file
5
test/input/mlrrc-with-profile-flag
Normal file
5
test/input/mlrrc-with-profile-flag
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
icsv
|
||||
profile j
|
||||
|
||||
[j]
|
||||
ojson
|
||||
9
test/input/mlrrc-with-profiles
Normal file
9
test/input/mlrrc-with-profiles
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
# Global settings, applied always:
|
||||
icsv
|
||||
|
||||
[j]
|
||||
ojson # Settings for the "j" profile
|
||||
jvstack
|
||||
|
||||
[p]
|
||||
opprint
|
||||
Loading…
Add table
Add a link
Reference in a new issue