Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-18 00:45:47 +00:00
Code Issues Projects Releases Wiki Activity

Record hashing perf (#781)

Browse source 
* todo

* Rename inputChannel,outputChannel to readerChannel,writerChannel

* Rename inputChannel,outputChannel to readerChannel,writerChannel (#772)

* Start batched-reader API mods

* Singleton-list step for reader-batching at input

* CLI options for records-per-batch and hash-records

* Push channelized-reader logic into DKVP reader

* Push batching logic into chain-transformer, transformers, and channel-writer

* foo

* cmd/mprof and cmd/mprof2

* cmd/mprof3 and cmd/mprof4

* narrowed in on regexp-splitting on IFS/IPS as perf-hit

* neaten

* channelize nidx

* cmd/mprof5

* channelize CSV reader

* channelize NIDX reader

* Dedupe DKVP-reader and NIDX-reader source files

* channelize CSV-lite reader

* channelize XTAB reader

* batchify JSON reader

* channelize GEN pseudo-reader

* scripts for perf-testing on larger files

* merge with main for #776

* Fix record-batching for join and repl

* Fix comment-handling in channelized XTAB reader

* Fix bug found in positional-rename

* Use --no-hash-records by default
This commit is contained in:
John Kerl 2021-12-14 22:41:40 -05:00 • committed by GitHub
parent f233923351
commit 58d9ad19bc
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
8 changed files with 67 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

14
docs/src/manpage.md
View file

@ -478,6 +478,14 @@ MISCELLANEOUS FLAGS
rather than after. May be used more than once.
Example: `mlr --from a.dat --from b.dat cat` is the
same as `mlr cat a.dat b.dat`.
--hash-records This is an internal parameter which normally does not
need to be modified. It controls the mechanism by
which Miller accesses fields within records. In
general --no-hash-records is faster, and is the
default. For specific use-cases involving data having
many fields, and many of them being processed during
a given processing run, --hash-records might offer a
slight performance benefit.
--infer-int-as-float or -A
Cast all integers in data files to floats.
--infer-no-octal or -O Treat numbers like 0123 in data files as string
@ -508,12 +516,16 @@ MISCELLANEOUS FLAGS
unlikely to be a noticeable performance improvement,
since direct-to-screen output for large files has its
own overhead.
--no-hash-records See --hash-records.
--nr-progress-mod {m} With m a positive integer: print filename and record
count to os.Stderr every m input records.
--ofmt {format} E.g. `%.18f`, `%.0f`, `%9.6e`. Please use
sprintf-style codes for floating-point nummbers. If
not specified, default formatting is used. See also
the `fmtnum` function and the `format-values` verb.
--records-per-batch {n} This is an internal parameter for maximum number of
records in a batch size. Normally this does not need
to be modified.
--seed {n} with `n` of the form `12345678` or `0xcafefeed`. For
`put`/`filter` `urand`, `urandint`, and `urand32`.
--tz {timezone} Specify timezone, overriding `$TZ` environment
@ -2994,5 +3006,5 @@ SEE ALSO
2021-12-07 MILLER(1)
2021-12-15 MILLER(1)
</pre>

14
docs/src/manpage.txt
View file

@ -457,6 +457,14 @@ MISCELLANEOUS FLAGS
rather than after. May be used more than once.
Example: `mlr --from a.dat --from b.dat cat` is the
same as `mlr cat a.dat b.dat`.
--hash-records This is an internal parameter which normally does not
need to be modified. It controls the mechanism by
which Miller accesses fields within records. In
general --no-hash-records is faster, and is the
default. For specific use-cases involving data having
many fields, and many of them being processed during
a given processing run, --hash-records might offer a
slight performance benefit.
--infer-int-as-float or -A
Cast all integers in data files to floats.
--infer-no-octal or -O Treat numbers like 0123 in data files as string
@ -487,12 +495,16 @@ MISCELLANEOUS FLAGS
unlikely to be a noticeable performance improvement,
since direct-to-screen output for large files has its
own overhead.
--no-hash-records See --hash-records.
--nr-progress-mod {m} With m a positive integer: print filename and record
count to os.Stderr every m input records.
--ofmt {format} E.g. `%.18f`, `%.0f`, `%9.6e`. Please use
sprintf-style codes for floating-point nummbers. If
not specified, default formatting is used. See also
the `fmtnum` function and the `format-values` verb.
--records-per-batch {n} This is an internal parameter for maximum number of
records in a batch size. Normally this does not need
to be modified.
--seed {n} with `n` of the form `12345678` or `0xcafefeed`. For
`put`/`filter` `urand`, `urandint`, and `urand32`.
--tz {timezone} Specify timezone, overriding `$TZ` environment
@ -2973,4 +2985,4 @@ SEE ALSO
2021-12-07 MILLER(1)
2021-12-15 MILLER(1)

6
docs/src/reference-main-flag-list.md
View file

@ -341,6 +341,8 @@ These are flags which don't fit into any other category.
`: Force buffered output to be written after every output record. The default is flush output after every record if the output is to the terminal, or less often if the output is to a file or a pipe. The default is a significant performance optimization for large files. Use this flag to force frequent updates even when output is to a pipe or file, at a performance cost.
* `--from {filename}
`: Use this to specify an input file before the verb(s), rather than after. May be used more than once. Example: `mlr --from a.dat --from b.dat cat` is the same as `mlr cat a.dat b.dat`.
* `--hash-records
`: This is an internal parameter which normally does not need to be modified. It controls the mechanism by which Miller accesses fields within records. In general --no-hash-records is faster, and is the default. For specific use-cases involving data having many fields, and many of them being processed during a given processing run, --hash-records might offer a slight performance benefit.
* `--infer-int-as-float or -A
`: Cast all integers in data files to floats.
* `--infer-no-octal or -O
@ -355,10 +357,14 @@ These are flags which don't fit into any other category.
`: Like `--load` but works with more than one filename, e.g. `--mload *.mlr --`.
* `--no-fflush
`: Let buffered output not be written after every output record. The default is flush output after every record if the output is to the terminal, or less often if the output is to a file or a pipe. The default is a significant performance optimization for large files. Use this flag to allow less-frequent updates when output is to the terminal. This is unlikely to be a noticeable performance improvement, since direct-to-screen output for large files has its own overhead.
* `--no-hash-records
`: See --hash-records.
* `--nr-progress-mod {m}
`: With m a positive integer: print filename and record count to os.Stderr every m input records.
* `--ofmt {format}
`: E.g. `%.18f`, `%.0f`, `%9.6e`. Please use sprintf-style codes for floating-point nummbers. If not specified, default formatting is used. See also the `fmtnum` function and the `format-values` verb.
* `--records-per-batch {n}
`: This is an internal parameter for maximum number of records in a batch size. Normally this does not need to be modified.
* `--seed {n}
`: with `n` of the form `12345678` or `0xcafefeed`. For `put`/`filter` `urand`, `urandint`, and `urand32`.
* `--tz {timezone}

8
internal/pkg/cli/option_parse.go
View file

@ -2578,7 +2578,11 @@ this does not need to be modified.`,
{
name: "--hash-records",
help: `This is an internal parameter which normally does not need to be modified.`,
help: `This is an internal parameter which normally does not need to be modified.
It controls the mechanism by which Miller accesses fields within records.
In general --no-hash-records is faster, and is the default. For specific use-cases involving
data having many fields, and many of them being processed during a given processing run,
--hash-records might offer a slight performance benefit.`,
parser: func(args []string, argc int, pargi *int, options *TOptions) {
types.HashRecords(true)
*pargi += 1
@ -2587,7 +2591,7 @@ this does not need to be modified.`,
{
name: "--no-hash-records",
help: `This is an internal parameter which normally does not need to be modified.`,
help: `See --hash-records.`,
parser: func(args []string, argc int, pargi *int, options *TOptions) {
types.HashRecords(false)
*pargi += 1

2
internal/pkg/types/mlrmap.go
View file

@ -58,7 +58,7 @@ package types
// Both these figures are for just doing mlr cat. At the moment I'm leaving this
// default-on pending more profiling on more complex record-processing operations
// such as mlr sort.
var hashRecords = true
var hashRecords = false
func HashRecords(onOff bool) {
hashRecords = onOff

14
man/manpage.txt
View file

@ -457,6 +457,14 @@ MISCELLANEOUS FLAGS
rather than after. May be used more than once.
Example: `mlr --from a.dat --from b.dat cat` is the
same as `mlr cat a.dat b.dat`.
--hash-records This is an internal parameter which normally does not
need to be modified. It controls the mechanism by
which Miller accesses fields within records. In
general --no-hash-records is faster, and is the
default. For specific use-cases involving data having
many fields, and many of them being processed during
a given processing run, --hash-records might offer a
slight performance benefit.
--infer-int-as-float or -A
Cast all integers in data files to floats.
--infer-no-octal or -O Treat numbers like 0123 in data files as string
@ -487,12 +495,16 @@ MISCELLANEOUS FLAGS
unlikely to be a noticeable performance improvement,
since direct-to-screen output for large files has its
own overhead.
--no-hash-records See --hash-records.
--nr-progress-mod {m} With m a positive integer: print filename and record
count to os.Stderr every m input records.
--ofmt {format} E.g. `%.18f`, `%.0f`, `%9.6e`. Please use
sprintf-style codes for floating-point nummbers. If
not specified, default formatting is used. See also
the `fmtnum` function and the `format-values` verb.
--records-per-batch {n} This is an internal parameter for maximum number of
records in a batch size. Normally this does not need
to be modified.
--seed {n} with `n` of the form `12345678` or `0xcafefeed`. For
`put`/`filter` `urand`, `urandint`, and `urand32`.
--tz {timezone} Specify timezone, overriding `$TZ` environment
@ -2973,4 +2985,4 @@ SEE ALSO
2021-12-07 MILLER(1)
2021-12-15 MILLER(1)

16
man/mlr.1
View file

@ -2,12 +2,12 @@
.\" Title: mlr
.\" Author: [see the "AUTHOR" section]
.\" Generator: ./mkman.rb
.\" Date: 2021-12-07
.\" Date: 2021-12-15
.\" Manual: \ \&
.\" Source: \ \&
.\" Language: English
.\"
.TH "MILLER" "1" "2021-12-07" "\ \&" "\ \&"
.TH "MILLER" "1" "2021-12-15" "\ \&" "\ \&"
.\" -----------------------------------------------------------------
.\" * Portability definitions
.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -576,6 +576,14 @@ These are flags which don't fit into any other category.
rather than after. May be used more than once.
Example: `mlr --from a.dat --from b.dat cat` is the
same as `mlr cat a.dat b.dat`.
--hash-records This is an internal parameter which normally does not
need to be modified. It controls the mechanism by
which Miller accesses fields within records. In
general --no-hash-records is faster, and is the
default. For specific use-cases involving data having
many fields, and many of them being processed during
a given processing run, --hash-records might offer a
slight performance benefit.
--infer-int-as-float or -A
Cast all integers in data files to floats.
--infer-no-octal or -O Treat numbers like 0123 in data files as string
@ -606,12 +614,16 @@ These are flags which don't fit into any other category.
unlikely to be a noticeable performance improvement,
since direct-to-screen output for large files has its
own overhead.
--no-hash-records See --hash-records.
--nr-progress-mod {m} With m a positive integer: print filename and record
count to os.Stderr every m input records.
--ofmt {format} E.g. `%.18f`, `%.0f`, `%9.6e`. Please use
sprintf-style codes for floating-point nummbers. If
not specified, default formatting is used. See also
the `fmtnum` function and the `format-values` verb.
--records-per-batch {n} This is an internal parameter for maximum number of
records in a batch size. Normally this does not need
to be modified.
--seed {n} with `n` of the form `12345678` or `0xcafefeed`. For
`put`/`filter` `urand`, `urandint`, and `urand32`.
--tz {timezone} Specify timezone, overriding `$TZ` environment

1
todo.txt
View file

@ -1,6 +1,7 @@
================================================================
PUNCHDOWN LIST
* --ifs-regex & --ips-regex -- guessing is not safe as evidence by '.' and '|'
* big-picture item @ Rmd; also webdoc intro page