Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-25 17:04:01 +00:00
Code Issues Projects Releases Wiki Activity
7876 commits 53 branches 76 tags 252 MiB
Commit graph

7876 commits

This branch
This branch
All branches
Author SHA1 Message Date
John Kerl
d4b67d07cd Merge branch 'jit-type-infer' 2021-12-20 23:58:05 -05:00
John Kerl
7a97c9b868
Performance improvement by JIT type inference (#786) 
* JIT mlrval type-interfence: mlrval package

* mlrmap refactor

* complete merge from #779

* iterating

* mlrval/format.go

* mlrval/copy.go

* bifs/arithmetic_test.go

* iterate on bifs/collections_test.go

* mlrval_cmp.go

* mlrval JSON iterate

* iterate applying mlrval refactors to dependent packages

* first clean compile in a long while on this branch

* results of first post-compile profiling

* testing

* bugfix in ofmt formatting

* bugfix in octal-supporess

* go fmt

* neaten

* regression tests all passing
 2021-12-20 23:56:04 -05:00
John Kerl
79b5bc1180 regression tests all passing 2021-12-20 23:25:36 -05:00
John Kerl
30bcb69e2e neaten 2021-12-20 22:44:44 -05:00
John Kerl
fc665c5b0d go fmt 2021-12-20 22:17:21 -05:00
John Kerl
6274e61126 bugfix in octal-supporess 2021-12-20 22:12:39 -05:00
John Kerl
5ba8d494f8 bugfix in ofmt formatting 2021-12-20 18:19:12 -05:00
John Kerl
4f9eae7dfa testing 2021-12-19 14:04:27 -05:00
John Kerl
9a0b959d48 results of first post-compile profiling 2021-12-17 17:35:43 -05:00
John Kerl
cbe8ee84e4 first clean compile in a long while on this branch 2021-12-17 16:48:01 -05:00
John Kerl
d33682073c iterate applying mlrval refactors to dependent packages 2021-12-16 18:30:34 -05:00
John Kerl
e84897bb75 mlrval JSON iterate 2021-12-16 17:52:10 -05:00
John Kerl
f18983c221 mlrval_cmp.go 2021-12-16 17:42:10 -05:00
John Kerl
6bcfe84ab4 iterate on bifs/collections_test.go 2021-12-15 09:26:40 -05:00
John Kerl
511a1bf3ee bifs/arithmetic_test.go 2021-12-14 22:47:02 -05:00
John Kerl
012e707917 mlrval/copy.go 2021-12-14 22:47:02 -05:00
John Kerl
1d3a918f7a mlrval/format.go 2021-12-14 22:47:02 -05:00
John Kerl
f8fd0da347 iterating 2021-12-14 22:47:02 -05:00
John Kerl
6773f6770e complete merge from #779 2021-12-14 22:47:00 -05:00
John Kerl
1f191cb7cb mlrmap refactor 2021-12-14 22:46:30 -05:00
John Kerl
eae73fe7cf JIT mlrval type-interfence: mlrval package 2021-12-14 22:46:30 -05:00
John Kerl
105a75fb30 Merge branch 'main' of git+ssh://github.com/johnkerl/miller 2021-12-14 22:43:17 -05:00
John Kerl
58d9ad19bc
Record hashing perf (#781) 
* todo

* Rename inputChannel,outputChannel to readerChannel,writerChannel

* Rename inputChannel,outputChannel to readerChannel,writerChannel (#772)

* Start batched-reader API mods

* Singleton-list step for reader-batching at input

* CLI options for records-per-batch and hash-records

* Push channelized-reader logic into DKVP reader

* Push batching logic into chain-transformer, transformers, and channel-writer

* foo

* cmd/mprof and cmd/mprof2

* cmd/mprof3 and cmd/mprof4

* narrowed in on regexp-splitting on IFS/IPS as perf-hit

* neaten

* channelize nidx

* cmd/mprof5

* channelize CSV reader

* channelize NIDX reader

* Dedupe DKVP-reader and NIDX-reader source files

* channelize CSV-lite reader

* channelize XTAB reader

* batchify JSON reader

* channelize GEN pseudo-reader

* scripts for perf-testing on larger files

* merge with main for #776

* Fix record-batching for join and repl

* Fix comment-handling in channelized XTAB reader

* Fix bug found in positional-rename

* Use --no-hash-records by default
 2021-12-14 22:41:40 -05:00
John Kerl
ba6286ca24 merge 2021-12-13 01:01:22 -05:00
John Kerl
641ecbdd13 merge 2021-12-13 01:00:46 -05:00
John Kerl
f233923351
Performance improvement: record-batching (#779) 
* Rename inputChannel,outputChannel to readerChannel,writerChannel

* Rename inputChannel,outputChannel to readerChannel,writerChannel (#772)

* Start batched-reader API mods

* Singleton-list step for reader-batching at input

* CLI options for records-per-batch and hash-records

* Push channelized-reader logic into DKVP reader

* Push batching logic into chain-transformer, transformers, and channel-writer

* foo

* cmd/mprof and cmd/mprof2

* cmd/mprof3 and cmd/mprof4

* narrowed in on regexp-splitting on IFS/IPS as perf-hit

* neaten

* channelize nidx

* cmd/mprof5

* channelize CSV reader

* channelize NIDX reader

* Dedupe DKVP-reader and NIDX-reader source files

* channelize CSV-lite reader

* channelize XTAB reader

* batchify JSON reader

* channelize GEN pseudo-reader

* scripts for perf-testing on larger files

* merge with main for #776

* Fix record-batching for join and repl

* Fix comment-handling in channelized XTAB reader

* Fix bug found in positional-rename
 2021-12-13 00:57:52 -05:00
John Kerl
fe6d6cfd4e Fix bug found in positional-rename 2021-12-13 00:42:51 -05:00
John Kerl
8cea6ae70a Fix comment-handling in channelized XTAB reader 2021-12-13 00:23:24 -05:00
John Kerl
056b8e973f Fix record-batching for join and repl 2021-12-13 00:10:35 -05:00
John Kerl
4aab213ecd merge with main for #776 2021-12-12 23:37:03 -05:00
John Kerl
34627e72a5 scripts for perf-testing on larger files 2021-12-12 23:37:03 -05:00
John Kerl
e24a36da63 channelize GEN pseudo-reader 2021-12-12 23:37:03 -05:00
John Kerl
f47b2dd2f0 batchify JSON reader 2021-12-12 23:37:03 -05:00
John Kerl
8403549149 channelize XTAB reader 2021-12-12 23:37:03 -05:00
John Kerl
be3286f606 channelize CSV-lite reader 2021-12-12 23:37:01 -05:00
John Kerl
4df477f734 Dedupe DKVP-reader and NIDX-reader source files 2021-12-12 23:36:16 -05:00
John Kerl
2618d63e6a channelize NIDX reader 2021-12-12 23:35:54 -05:00
John Kerl
d183d1191c channelize CSV reader 2021-12-12 23:35:54 -05:00
John Kerl
79bc7fdc20 cmd/mprof5 2021-12-12 23:35:54 -05:00
John Kerl
5666b599a7 channelize nidx 2021-12-12 23:35:54 -05:00
John Kerl
3faf66af21 neaten 2021-12-12 23:35:54 -05:00
John Kerl
b6ab5d982f narrowed in on regexp-splitting on IFS/IPS as perf-hit 2021-12-12 23:35:54 -05:00
John Kerl
6ad475bfc0 cmd/mprof3 and cmd/mprof4 2021-12-12 23:35:53 -05:00
John Kerl
7f1aced97b cmd/mprof and cmd/mprof2 2021-12-12 23:35:53 -05:00
John Kerl
0335c4e70f foo 2021-12-12 23:35:53 -05:00
John Kerl
9bdc53d783 Push batching logic into chain-transformer, transformers, and channel-writer 2021-12-12 23:35:53 -05:00
John Kerl
aaf0c27a6a Push channelized-reader logic into DKVP reader 2021-12-12 23:35:53 -05:00
John Kerl
5c022758e9 CLI options for records-per-batch and hash-records 2021-12-12 23:35:53 -05:00
John Kerl
f2879aebde Singleton-list step for reader-batching at input 2021-12-12 23:35:53 -05:00
John Kerl
2f8d2b18f9 Start batched-reader API mods 2021-12-12 23:35:53 -05:00