Commit graph

42 commits

Author SHA1 Message Date
John Kerl
6f22401d53
Exclude median from summary default (#1131) 2022-11-26 10:34:28 -05:00
John Kerl
63db1a9059
mlr reshape multiple regex via multiple -r (#1091) 2022-09-05 22:05:54 -04:00
John Kerl
4a53a6b278
Fix labels for mlr histogram --auto (#1089) 2022-09-03 09:55:44 -04:00
John Kerl
dafec6dc09
mlr cat --filename / --filenum (#1080)
* mlr cat --filename / --filenum

* codespell
2022-08-20 08:42:15 -04:00
John Kerl
a64133250c
Fix natsort of empty strings; support mlr sort -rt same as -tr (#1068)
* mlr sort -rt == mlrt sort -tr

* fix natsort of empty strings
2022-08-06 23:10:09 -04:00
John Kerl
8bca869b68 update null_count for use in stats1 2022-07-12 01:27:04 -04:00
John Kerl
64b3cbfcfe
mlr summary verb (#1056)
* mlr summary verb

* doc content

* iterating on more summarizers; needs work as noted in the PR

* mlr summary with selectable accumulators

* code-review feedback

* iterating on transposed output

* extend unit-test cases

* doc extensions
2022-07-12 00:57:34 -04:00
John Kerl
6b32e1f419
Reduce number of os.Exit callsites, part 1 of n (#1055)
* neaten some uses of os.Exit

* iterating

* iterating

* iterating

* iterating

* iterating

* iterating

* iterating
2022-07-09 18:53:24 -04:00
John Kerl
52ce2473fb
Fix issue 1037 (#1047)
* Fix issue 1037

* test files
2022-07-04 15:40:42 -04:00
John Kerl
de9dbfc212 Fix panic on 'mlr sort -n' 2022-03-28 23:26:37 -04:00
John Kerl
2408915160
DSL functions and verbs for UTF-8 <-> Latin-1 (#997)
* latin1_to_utf8 and utf8_to_latin1 DSL functions

* doc-build artifacts for previous commit

* Test cases for latin1_to_utf8 and utf8_to_latin1

* extend on-line help

* latin1_to_utf8 and utf8_to_latin1 verbs

* unit-test cases for verbs

* Keep with kebab-case naming convention for verbs

* webdocs
2022-03-20 17:29:40 -04:00
John Kerl
4191c35b7c
Re-use whitespace regexp in clean_whitespace (#994)
* DSL strict mode

* Re-use whitespace regexp in clean_whitespace
2022-03-20 12:17:51 -04:00
John Kerl
9b806f9561
Add --left-keep-fields option for mlr join (#967)
* Add --left-keep-fields option for mlr join

* on-line help for mlr join --lk

* doc-build artifacts

* test cases
2022-02-23 23:32:19 -05:00
John Kerl
228f73415e
Add --implicit-tsv-header as alias for --implicit-csv-header, etc (#952)
* Add -implicit-tsv-header as aliias for --implicit-csv-header, etc

* doc-build artifacts for previous commit
2022-02-21 12:48:43 -05:00
John Kerl
e1c2a7a07a Remove some dead code 2022-02-18 22:40:27 -05:00
John Kerl
f444459b25
Fix handling of mlr nest abbrevs (#937)
* Fix handling of mlr nest abbrevs

* expanded test cases
2022-02-10 10:17:39 -05:00
John Kerl
ca9505dfaf
Natural sort (#932)
* Add natural sort order as an option for the sort verb

* Add natural sort order as an option for the sort DSL function

* doc-build artifacts for on-line help

* webdocs

* codespell fix

* unit-test files for sort verb

* unit-test files for sort DSL function
2022-02-08 00:35:28 -05:00
Stephen Kitt
1beacbd381
More dead code removal (#905)
Signed-off-by: Stephen Kitt <steve@sk2.org>
2022-01-30 12:03:41 -05:00
Stephen Kitt
d536318ed6
Use int64 wherever "64-bit integer" is assumed (#902)
Miller assumes 64-bit integers, but in Go, the int type varies in size
depending on the architecture: 32-bit architectures have int
equivalent to int32. As a result, the supported range of integer
values is greatly reduced on 32-bit architectures compared to what is
suggested by the documentation.

This patch explicitly uses int64 wherever 64-bit integers are
assumed.

Test cases affected by the behaviour of the random generator are
updated to reflect the new values (the existing seed doesn't produce
the same behaviour since the way random values are generated has
changed).

Signed-off-by: Stephen Kitt <steve@sk2.org>
2022-01-27 12:06:25 -05:00
John Kerl
f7ff63124b
mlr split verb (#898)
* mlr split

* regression-test cases

* doc-build artifacts
2022-01-26 23:16:25 -05:00
John Kerl
62a50fb9a5 Fix slwin-7-2 -> slwin_7_2 doc info 2022-01-24 16:33:05 -05:00
John Kerl
dedc7ff06f replace - with _ in *.go filenames 2022-01-23 23:22:55 -05:00
John Kerl
b4c50470a2 typofix 2022-01-23 23:18:23 -05:00
John Kerl
77811a4788
Sliding window averages (#894)
* todo

* Neaten existing DSL sketch

* rebase on #893, and sketch

* code-complete

* build artifacts for previous commit

* replace - with _ in shift and slwin
2022-01-23 23:03:46 -05:00
John Kerl
a2a9118ad8
Implement shift-lead option for mlr step (#893)
* iterating

* stepper-input refactor in prep for sliding-window PR

* window-keeper util class

* integrate window-keeper into step-transformer
2022-01-23 00:54:39 -05:00
John Kerl
eb7e29d207 neaten 2022-01-17 22:57:19 -05:00
John Kerl
471a110e85
Mlrval arrayval from []Mlrval to []*Mlrval (#880)
* to do
* Mlrval arrayval from []Mlrval to []*Mlrval
2022-01-17 15:13:57 -05:00
John Kerl
d25cf6fd70
Improve mlr top documentation for #861 (#875)
* Doc improvements for mlr top -a

* Build-artifacts for previous commit
2022-01-16 17:00:28 -05:00
John Kerl
0c75713b94
Fix codespell config to skip multiple files (#837)
* trying to get codespell to trigger before merge to main

* iterating

* https://github.com/codespell-project/actions-codespell

* iterating

* iterating

* iterating

* iterating

* iterating

* more

* more

* more
2022-01-03 23:16:43 -05:00
John Kerl
be23208a1a Fix mlr tail -n4 2022-01-03 21:15:56 -05:00
John Kerl
a977617797
Address some staticcheck issues (#823)
* address some staticcheck issues

* address some staticcheck issues

* address some staticcheck issues

* address some staticcheck issues
2022-01-01 14:28:19 -05:00
John Kerl
c0e8fcb811
Simplify and unit-test cmp disposition matrices (#819)
* change version from 6.0.0-dev to 6.0.0-rc

* Simplify and unit-test cmp-matrices
2021-12-29 21:53:56 -05:00
John Kerl
096bb9bc12
Make --ifs-regex and --ips-regex explicit command-line flags (#799)
* Function-pointerize IXS/IXSRegex to reduce runtime iffelsing

* remove IsRegexString and SuppressIXSRegex

* regression tests passing

* doc updates
2021-12-25 00:00:18 -05:00
John Kerl
7a97c9b868
Performance improvement by JIT type inference (#786)
* JIT mlrval type-interfence: mlrval package

* mlrmap refactor

* complete merge from #779

* iterating

* mlrval/format.go

* mlrval/copy.go

* bifs/arithmetic_test.go

* iterate on bifs/collections_test.go

* mlrval_cmp.go

* mlrval JSON iterate

* iterate applying mlrval refactors to dependent packages

* first clean compile in a long while on this branch

* results of first post-compile profiling

* testing

* bugfix in ofmt formatting

* bugfix in octal-supporess

* go fmt

* neaten

* regression tests all passing
2021-12-20 23:56:04 -05:00
John Kerl
f233923351
Performance improvement: record-batching (#779)
* Rename inputChannel,outputChannel to readerChannel,writerChannel

* Rename inputChannel,outputChannel to readerChannel,writerChannel (#772)

* Start batched-reader API mods

* Singleton-list step for reader-batching at input

* CLI options for records-per-batch and hash-records

* Push channelized-reader logic into DKVP reader

* Push batching logic into chain-transformer, transformers, and channel-writer

* foo

* cmd/mprof and cmd/mprof2

* cmd/mprof3 and cmd/mprof4

* narrowed in on regexp-splitting on IFS/IPS as perf-hit

* neaten

* channelize nidx

* cmd/mprof5

* channelize CSV reader

* channelize NIDX reader

* Dedupe DKVP-reader and NIDX-reader source files

* channelize CSV-lite reader

* channelize XTAB reader

* batchify JSON reader

* channelize GEN pseudo-reader

* scripts for perf-testing on larger files

* merge with main for #776

* Fix record-batching for join and repl

* Fix comment-handling in channelized XTAB reader

* Fix bug found in positional-rename
2021-12-13 00:57:52 -05:00
John Kerl
4ddb8ff25c Rename inputChannel,outputChannel to readerChannel,writerChannel (#772) 2021-12-07 08:02:30 -05:00
John Kerl
4c3e141df8
Split separators from context (#771)
* Fix mlr -O to return strings for all octal numbers along with 0* numbers like 07 and 08

* Factor separators out of Context struct
2021-12-06 22:47:58 -05:00
John Kerl
d9dbb1e92e
Improvements for shebang-script handling (#758)
* Experimental #!/usr/bin/env mlr -s feature

* Allow flags after verbs, for shebang support
2021-11-24 01:15:28 -05:00
John Kerl
4394819cf8
Approximate-match feature for online help (#754)
* Explicitly support approximate-match help

* Unit tests for exact and approximate help
2021-11-17 23:37:48 -05:00
John Kerl
b14c917a79
Neaten output of mlr stats1 --help (#751) 2021-11-15 07:46:11 -05:00
John Kerl
bc72cd1857
More Go-package restructuring (#748) 2021-11-12 12:49:55 -05:00
John Kerl
e2b6ec2391
Standardize Go-package structure (#746) 2021-11-11 14:15:13 -05:00