miller/internal/pkg/transformers
John Kerl f233923351
Performance improvement: record-batching (#779)
* Rename inputChannel,outputChannel to readerChannel,writerChannel

* Rename inputChannel,outputChannel to readerChannel,writerChannel (#772)

* Start batched-reader API mods

* Singleton-list step for reader-batching at input

* CLI options for records-per-batch and hash-records

* Push channelized-reader logic into DKVP reader

* Push batching logic into chain-transformer, transformers, and channel-writer

* foo

* cmd/mprof and cmd/mprof2

* cmd/mprof3 and cmd/mprof4

* narrowed in on regexp-splitting on IFS/IPS as perf-hit

* neaten

* channelize nidx

* cmd/mprof5

* channelize CSV reader

* channelize NIDX reader

* Dedupe DKVP-reader and NIDX-reader source files

* channelize CSV-lite reader

* channelize XTAB reader

* batchify JSON reader

* channelize GEN pseudo-reader

* scripts for perf-testing on larger files

* merge with main for #776

* Fix record-batching for join and repl

* Fix comment-handling in channelized XTAB reader

* Fix bug found in positional-rename
2021-12-13 00:57:52 -05:00
..
utils Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
aaa_chain_transformer.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
aaa_record_transformer.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
aaa_transformer_table.go Approximate-match feature for online help (#754) 2021-11-17 23:37:48 -05:00
altkv.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
bar.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
bootstrap.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
cat.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
check.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
clean-whitespace.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
count-similar.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
count.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
cut.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
decimate.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
doc.go Standardize Go-package structure (#746) 2021-11-11 14:15:13 -05:00
fill-down.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
fill-empty.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
flatten.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
format-values.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
fraction.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
gap.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
grep.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
group-by.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
group-like.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
having-fields.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
head.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
histogram.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
join.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
json-parse.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
json-stringify.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
label.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
merge-fields.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
most-or-least-frequent.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
nest.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
nothing.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
put-or-filter.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
README.md Standardize Go-package structure (#746) 2021-11-11 14:15:13 -05:00
regularize.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
remove-empty-columns.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
rename.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
reorder.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
repeat.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
reshape.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
sample.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
sec2gmt.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
sec2gmtdate.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
seqgen.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
shuffle.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
skip-trivial-records.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
sort-within-records.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
sort.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
stats1.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
stats2.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
step.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
tac.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
tail.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
tee.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
template.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
top.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
unflatten.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
uniq.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00
unsparsify.go Performance improvement: record-batching (#779) 2021-12-13 00:57:52 -05:00

Logic for transforming input records into output records as requested by the user (sort, filter, etc.).:

  • The IRecordTransformer abstract record-transformer interface datatype, as well as the ChainTransformer Go-channel chaining mechanism for piping one transformer into the next.
  • The transformer lookup table, used for Miller command-line parsing, verb construction, and online help.
  • All the concrete record-transformers such as cat, tac, sort, put, and so on.