Commit graph

42 commits

Author SHA1 Message Date
John Kerl
8267b2220d 6.0.0.rc1 2022-01-08 15:47:19 -05:00
John Kerl
0c75713b94
Fix codespell config to skip multiple files (#837)
* trying to get codespell to trigger before merge to main

* iterating

* https://github.com/codespell-project/actions-codespell

* iterating

* iterating

* iterating

* iterating

* iterating

* more

* more

* more
2022-01-03 23:16:43 -05:00
John Kerl
1b9526e585
More codespell fixes (#834)
* Fix mlr tail -n4

* More codespell fixes
2022-01-03 21:40:53 -05:00
John Kerl
a977617797
Address some staticcheck issues (#823)
* address some staticcheck issues

* address some staticcheck issues

* address some staticcheck issues

* address some staticcheck issues
2022-01-01 14:28:19 -05:00
John Kerl
c1a8b0d230
allow substr to string-cast; helpful for ZIP codes etc. (#820) 2021-12-30 12:00:20 -05:00
John Kerl
c0e8fcb811
Simplify and unit-test cmp disposition matrices (#819)
* change version from 6.0.0-dev to 6.0.0-rc

* Simplify and unit-test cmp-matrices
2021-12-29 21:53:56 -05:00
John Kerl
db866f5e02 change version from 6.0.0-dev to 6.0.0-rc 2021-12-29 21:40:51 -05:00
John Kerl
3907d749f8 go fmt 2021-12-29 21:32:00 -05:00
John Kerl
344b717064
Re-index mvtype to match sort-order (#815)
* todo

* Re-index mvtype to match sort-order
2021-12-29 15:26:00 -05:00
John Kerl
e10fee0724
Improve type-inference performance (#809)
* To-do items for broader platform/go-version benchmarking

* neaten inferrer API

* extend type-inference unit-test cases

* Add benchmark scripts for comparing compiler versions

* mlr version in addition to mlr --version

* some go-benchmark files for Mac/Linux perf comparisons

* neaten perf-scripts

* merge

* type-scan optimization tests

* type-scan optimization infra

* test new inferrer

* mlr --time option

* include --cpuprofile and --traceprofile in on-line help

* sharpen inferred/deferred-type API distinction

* replace old inferrer with newer/faster

* update docs for new type-inferrer
2021-12-27 00:54:21 -05:00
John Kerl
5e8d3fddd0
Suppress read of .mlrrc during regression tests (#808) 2021-12-26 10:53:19 -05:00
John Kerl
25fb7f1d78
Document --norc flag (#806) 2021-12-25 13:11:58 -05:00
John Kerl
f56ab87c2d
Docs for funct keyword (#805) 2021-12-25 12:58:02 -05:00
John Kerl
984df274bb Minor neatens 2021-12-25 12:16:10 -05:00
John Kerl
bca738a781
Work around Go strconv's allowance of underscores in int/float literals (#801)
* Function-pointerize IXS/IXSRegex to reduce runtime iffelsing

* remove IsRegexString and SuppressIXSRegex

* regression tests passing

* doc updates

* Don't use sed -I in ./configure

* Work around Go strconv's allowance of _ in int/float literals
2021-12-25 00:38:13 -05:00
John Kerl
096bb9bc12
Make --ifs-regex and --ips-regex explicit command-line flags (#799)
* Function-pointerize IXS/IXSRegex to reduce runtime iffelsing

* remove IsRegexString and SuppressIXSRegex

* regression tests passing

* doc updates
2021-12-25 00:00:18 -05:00
John Kerl
9c8f8680d6 Fix minor doc-formatting issue on PR #794 2021-12-22 21:29:39 -05:00
John Kerl
157e567909
Dedupe field names by default (#794) 2021-12-22 21:07:29 -05:00
John Kerl
6b87a121b0 Copy-reduction in CSV/CSV-lite record-readers; very minor perf improvement 2021-12-22 20:01:16 -05:00
John Kerl
1d3bc11b29
urandelement DSL function (#789) 2021-12-21 23:15:14 -05:00
John Kerl
99e4a23ac6 typofix in regtest 2021-12-21 22:55:50 -05:00
John Kerl
93862f16f9
update mlr -O behavior for #756 (#788) 2021-12-21 22:40:34 -05:00
John Kerl
2a2ff7a768
Copy-reduction performance in Mlrval.String() (#787) 2021-12-21 09:53:03 -05:00
John Kerl
7a97c9b868
Performance improvement by JIT type inference (#786)
* JIT mlrval type-interfence: mlrval package

* mlrmap refactor

* complete merge from #779

* iterating

* mlrval/format.go

* mlrval/copy.go

* bifs/arithmetic_test.go

* iterate on bifs/collections_test.go

* mlrval_cmp.go

* mlrval JSON iterate

* iterate applying mlrval refactors to dependent packages

* first clean compile in a long while on this branch

* results of first post-compile profiling

* testing

* bugfix in ofmt formatting

* bugfix in octal-supporess

* go fmt

* neaten

* regression tests all passing
2021-12-20 23:56:04 -05:00
John Kerl
58d9ad19bc
Record hashing perf (#781)
* todo

* Rename inputChannel,outputChannel to readerChannel,writerChannel

* Rename inputChannel,outputChannel to readerChannel,writerChannel (#772)

* Start batched-reader API mods

* Singleton-list step for reader-batching at input

* CLI options for records-per-batch and hash-records

* Push channelized-reader logic into DKVP reader

* Push batching logic into chain-transformer, transformers, and channel-writer

* foo

* cmd/mprof and cmd/mprof2

* cmd/mprof3 and cmd/mprof4

* narrowed in on regexp-splitting on IFS/IPS as perf-hit

* neaten

* channelize nidx

* cmd/mprof5

* channelize CSV reader

* channelize NIDX reader

* Dedupe DKVP-reader and NIDX-reader source files

* channelize CSV-lite reader

* channelize XTAB reader

* batchify JSON reader

* channelize GEN pseudo-reader

* scripts for perf-testing on larger files

* merge with main for #776

* Fix record-batching for join and repl

* Fix comment-handling in channelized XTAB reader

* Fix bug found in positional-rename

* Use --no-hash-records by default
2021-12-14 22:41:40 -05:00
John Kerl
f233923351
Performance improvement: record-batching (#779)
* Rename inputChannel,outputChannel to readerChannel,writerChannel

* Rename inputChannel,outputChannel to readerChannel,writerChannel (#772)

* Start batched-reader API mods

* Singleton-list step for reader-batching at input

* CLI options for records-per-batch and hash-records

* Push channelized-reader logic into DKVP reader

* Push batching logic into chain-transformer, transformers, and channel-writer

* foo

* cmd/mprof and cmd/mprof2

* cmd/mprof3 and cmd/mprof4

* narrowed in on regexp-splitting on IFS/IPS as perf-hit

* neaten

* channelize nidx

* cmd/mprof5

* channelize CSV reader

* channelize NIDX reader

* Dedupe DKVP-reader and NIDX-reader source files

* channelize CSV-lite reader

* channelize XTAB reader

* batchify JSON reader

* channelize GEN pseudo-reader

* scripts for perf-testing on larger files

* merge with main for #776

* Fix record-batching for join and repl

* Fix comment-handling in channelized XTAB reader

* Fix bug found in positional-rename
2021-12-13 00:57:52 -05:00
John Kerl
7c9cc61ac9
Fix issue with pipe or dot as IFS for DKVP/NIDX/CSV-lite (#778) 2021-12-12 23:25:16 -05:00
John Kerl
107ab5c4c9
Performance improvement for CSV-lite and DKVP (#774)
* Clarify build-from-source steps

* Performance improvement for CSV-lite and DKVP
2021-12-08 22:47:53 -05:00
John Kerl
4ddb8ff25c Rename inputChannel,outputChannel to readerChannel,writerChannel (#772) 2021-12-07 08:02:30 -05:00
John Kerl
4c3e141df8
Split separators from context (#771)
* Fix mlr -O to return strings for all octal numbers along with 0* numbers like 07 and 08

* Factor separators out of Context struct
2021-12-06 22:47:58 -05:00
John Kerl
8e975c9d39
mlr -O regexp bugfix (#767) 2021-12-03 11:24:19 -05:00
John Kerl
7642f7ede6
Fix mlr -O to return strings for all octal numbers along with 0* numbers like 07 and 08 (#766) 2021-12-01 21:18:19 -05:00
John Kerl
cef753c232
Performance improvement: buffered output (#765)
* Experimental #!/usr/bin/env mlr -s feature

* Allow flags after verbs, for shebang support

* Performance optmization: less frequent syscall.write on output
2021-12-01 20:12:31 -05:00
John Kerl
c449c32c39 Fix auto-insert of -- for mlr -s feature (forgotten on previous commit) 2021-11-24 18:21:24 -05:00
John Kerl
d9dbb1e92e
Improvements for shebang-script handling (#758)
* Experimental #!/usr/bin/env mlr -s feature

* Allow flags after verbs, for shebang support
2021-11-24 01:15:28 -05:00
John Kerl
b7e9c50444 merge 2021-11-17 23:40:02 -05:00
John Kerl
a73089be20 mlr --csv --irs/--ors crlf is OK, sine autodetected per platform 2021-11-17 23:39:16 -05:00
John Kerl
4394819cf8
Approximate-match feature for online help (#754)
* Explicitly support approximate-match help

* Unit tests for exact and approximate help
2021-11-17 23:37:48 -05:00
John Kerl
b14c917a79
Neaten output of mlr stats1 --help (#751) 2021-11-15 07:46:11 -05:00
John Kerl
dba3ed5dc4 Reorganization of new-in-miller-6 docpage (#750) 2021-11-13 23:18:35 -05:00
John Kerl
bc72cd1857
More Go-package restructuring (#748) 2021-11-12 12:49:55 -05:00
John Kerl
e2b6ec2391
Standardize Go-package structure (#746) 2021-11-11 14:15:13 -05:00