Commit graph

58 commits

Author SHA1 Message Date
John Kerl
8f04d7671d
Restore --tsvlite (#984)
* Restore --tsvlite flag

* todo

* doc-build artifacts

* doc note on --tsv vs --tsvlite and backslashed data
2022-03-15 09:01:20 -04:00
John Kerl
47ef5de0bb
Let dhms2sec accept input like "8h" (#983)
* Let dhms2sec accept input like "8h"

* Regression-test cases
2022-03-14 23:47:50 -04:00
John Kerl
3aebb68ad6 test cases and doc-build artifacts 2022-02-24 09:57:34 -05:00
John Kerl
9b806f9561
Add --left-keep-fields option for mlr join (#967)
* Add --left-keep-fields option for mlr join

* on-line help for mlr join --lk

* doc-build artifacts

* test cases
2022-02-23 23:32:19 -05:00
John Kerl
bfd2ec14c0 Fix test cases to match the new reality 2022-02-21 23:37:05 -05:00
John Kerl
763ad96b87 Fix test cases to match the new reality 2022-02-21 23:22:50 -05:00
John Kerl
228f73415e
Add --implicit-tsv-header as alias for --implicit-csv-header, etc (#952)
* Add -implicit-tsv-header as aliias for --implicit-csv-header, etc

* doc-build artifacts for previous commit
2022-02-21 12:48:43 -05:00
John Kerl
47a427b00c
Fix "%%" in strptime; more test cases for strptime (#951)
* strptime debug mode

* Fix strptime %% format code

* internal/pkg/pbnjay-strptime/strptime_test.go; reg-test files

* Web-doc improvements for strftime and strptime

* doc-build artifacts
2022-02-21 12:21:04 -05:00
John Kerl
bb95f37066
New fmtifnum DSL function; make fmtnum/fmtifnum recursive over maps and arrays (#946)
* New fmtifnum DSL function

* doc-build artifacts for previous commit

* Make fmtnum/fmtifnum recursive over maps and arrays

* Online-help examples for fmtnum/fmtifnum

* doc-build artifacts for previous commit

* unit-test cases

* code-dedupe
2022-02-14 23:57:14 -05:00
John Kerl
9d0159700c
Support %F, %T, and more in strptime (#944)
* Support %F, %T, and more in strptime

* Unit-test cases

* code-neaten
2022-02-13 19:38:17 -05:00
John Kerl
f444459b25
Fix handling of mlr nest abbrevs (#937)
* Fix handling of mlr nest abbrevs

* expanded test cases
2022-02-10 10:17:39 -05:00
John Kerl
d67868bd33
Add Inf and NaN literals to the DSL (#933)
* Add Inf and NaN literals to the DSL

* Parser-build artifacts

* is_nan DSL function

* doc-build artifacts

* webdoc material
2022-02-08 09:58:50 -05:00
John Kerl
ca9505dfaf
Natural sort (#932)
* Add natural sort order as an option for the sort verb

* Add natural sort order as an option for the sort DSL function

* doc-build artifacts for on-line help

* webdocs

* codespell fix

* unit-test files for sort verb

* unit-test files for sort DSL function
2022-02-08 00:35:28 -05:00
John Kerl
12e457bc96
strptime %j format for 3-digit day in year (#930)
* strptime %j format for 3-digit day in year

* doc mods
2022-02-07 09:10:27 -05:00
John Kerl
b983f825c0 Clarify time on-line help 2022-02-06 13:25:35 -05:00
John Kerl
9f3b1c8711
Fix is_non_empty for absent case (#928) 2022-02-06 13:10:36 -05:00
John Kerl
9f5a11f707
New --lazy-quotes flag for helping with malformed CSV (#925)
* Lazy-quotes option for CSV parser

* doc-build artifacts

* test cases

* doc-proofreads
2022-02-06 01:05:35 -05:00
John Kerl
66c4a077fd
Make TSV finally true TSV (#923)
* Spec-TSV

* doc mods; more test cases
2022-02-06 00:13:55 -05:00
John Kerl
9b61ccf05c fix merge 2022-01-30 14:14:01 -05:00
John Kerl
1a8d7edfd0 fix merge 2022-01-30 12:14:25 -05:00
John Kerl
971255ff2b typofix 2022-01-30 12:06:23 -05:00
Stephen Kitt
d536318ed6
Use int64 wherever "64-bit integer" is assumed (#902)
Miller assumes 64-bit integers, but in Go, the int type varies in size
depending on the architecture: 32-bit architectures have int
equivalent to int32. As a result, the supported range of integer
values is greatly reduced on 32-bit architectures compared to what is
suggested by the documentation.

This patch explicitly uses int64 wherever 64-bit integers are
assumed.

Test cases affected by the behaviour of the random generator are
updated to reflect the new values (the existing seed doesn't produce
the same behaviour since the way random values are generated has
changed).

Signed-off-by: Stephen Kitt <steve@sk2.org>
2022-01-27 12:06:25 -05:00
John Kerl
f7ff63124b
mlr split verb (#898)
* mlr split

* regression-test cases

* doc-build artifacts
2022-01-26 23:16:25 -05:00
John Kerl
62a50fb9a5 Fix slwin-7-2 -> slwin_7_2 doc info 2022-01-24 16:33:05 -05:00
John Kerl
b4c50470a2 typofix 2022-01-23 23:18:23 -05:00
John Kerl
77811a4788
Sliding window averages (#894)
* todo

* Neaten existing DSL sketch

* rebase on #893, and sketch

* code-complete

* build artifacts for previous commit

* replace - with _ in shift and slwin
2022-01-23 23:03:46 -05:00
John Kerl
a2a9118ad8
Implement shift-lead option for mlr step (#893)
* iterating

* stepper-input refactor in prep for sliding-window PR

* window-keeper util class

* integrate window-keeper into step-transformer
2022-01-23 00:54:39 -05:00
John Kerl
b6bad96caf
Fix #891 (#892)
* Fix #891

* update tests; artifacts for previous commit
2022-01-20 13:34:58 -05:00
John Kerl
4fb0b716ba
Make is_null/is_not_null DSL functions include new JSON-null type (#883)
* Make is_null/is_not_null DSL functions include new JSON-null type

* regression-test files
2022-01-18 08:25:03 -05:00
John Kerl
eb7e29d207 neaten 2022-01-17 22:57:19 -05:00
Stephen Kitt
af7bacf00e
Avoid assuming ./mlr is the mlr to test (#876)
In test cases, support a ${MLR} placeholder for the mlr binary under
test, instead of assuming it's ./mlr (or its equivalent on platforms
not using forward slashes).

This ensures that wrapped mlr invocations all use the same binary,
either the default or the binary specified by the user, avoiding
surprises when ./mlr doesn't exist or is a different version.

Signed-off-by: Stephen Kitt <steve@sk2.org>
2022-01-17 17:01:40 -05:00
John Kerl
d25cf6fd70
Improve mlr top documentation for #861 (#875)
* Doc improvements for mlr top -a

* Build-artifacts for previous commit
2022-01-16 17:00:28 -05:00
John Kerl
a25e34bab7 neaten 2022-01-13 10:34:11 -05:00
John Kerl
1785f8d111 doc-build artifacts for previous commit 2022-01-13 09:46:39 -05:00
John Kerl
e53dee0871 regression tests 2022-01-13 09:28:46 -05:00
John Kerl
ae722de887 neaten tree structure of a regression-test case 2022-01-12 22:44:05 -05:00
John Kerl
a0048f0393
New format DSL function (#869)
* New format DSL function

* Updated affected test cases involving on-line help on "for" prefix

* doc-build artifacts for previous commit

* regression-test cases
2022-01-12 22:40:59 -05:00
John Kerl
1e1e03e4b3
New concat DSL function for arrays (#868)
* New concat DSL function for arrays

* Documentation artifacts for previous commit

* Regression-test cases
2022-01-12 21:14:27 -05:00
John Kerl
67179eb791
Allow 0o... octal literals in the DSL (#864)
* Continue from #856

* Update expected-fail messages in test cases

* Allow 0o... octal literals in the DSL

* Build artifacts for previous commit

* go.mod/go.sum for goccmack/gocc

* Regression-test case

* Doc updates
2022-01-11 19:46:45 -05:00
John Kerl
b903227a5c
Continue #856 (#865)
* Continue from #856

* Update expected-fail messages in test cases
2022-01-11 19:42:08 -05:00
John Kerl
da91878939
Distinguish between JSON and JSON Lines formats (#844)
* Draw a distinction between JSON and JSON Lines formats

* Add JSON Lines to on-line help example

* Have JSON format default to --jlistwrap and --jvstack

* Update test cases for --jlistwrap output for JSON output format

* Have JSON format default to --jlistwrap and --jvstack for --{X}2j as well

* Make --jlistwrap / --jvstack as legacy flags, since now --json and --jsonl

* Add --c2l, --l2c, etc. command-line flags

* docmods for JSON Lines

* Update regression-test cases for JSON / JSON Lines distinction
2022-01-09 11:11:54 -05:00
John Kerl
a977617797
Address some staticcheck issues (#823)
* address some staticcheck issues

* address some staticcheck issues

* address some staticcheck issues

* address some staticcheck issues
2022-01-01 14:28:19 -05:00
John Kerl
c1a8b0d230
allow substr to string-cast; helpful for ZIP codes etc. (#820) 2021-12-30 12:00:20 -05:00
John Kerl
c0e8fcb811
Simplify and unit-test cmp disposition matrices (#819)
* change version from 6.0.0-dev to 6.0.0-rc

* Simplify and unit-test cmp-matrices
2021-12-29 21:53:56 -05:00
John Kerl
e10fee0724
Improve type-inference performance (#809)
* To-do items for broader platform/go-version benchmarking

* neaten inferrer API

* extend type-inference unit-test cases

* Add benchmark scripts for comparing compiler versions

* mlr version in addition to mlr --version

* some go-benchmark files for Mac/Linux perf comparisons

* neaten perf-scripts

* merge

* type-scan optimization tests

* type-scan optimization infra

* test new inferrer

* mlr --time option

* include --cpuprofile and --traceprofile in on-line help

* sharpen inferred/deferred-type API distinction

* replace old inferrer with newer/faster

* update docs for new type-inferrer
2021-12-27 00:54:21 -05:00
John Kerl
f56ab87c2d
Docs for funct keyword (#805) 2021-12-25 12:58:02 -05:00
John Kerl
096bb9bc12
Make --ifs-regex and --ips-regex explicit command-line flags (#799)
* Function-pointerize IXS/IXSRegex to reduce runtime iffelsing

* remove IsRegexString and SuppressIXSRegex

* regression tests passing

* doc updates
2021-12-25 00:00:18 -05:00
John Kerl
157e567909
Dedupe field names by default (#794) 2021-12-22 21:07:29 -05:00
John Kerl
99e4a23ac6 typofix in regtest 2021-12-21 22:55:50 -05:00
John Kerl
7a97c9b868
Performance improvement by JIT type inference (#786)
* JIT mlrval type-interfence: mlrval package

* mlrmap refactor

* complete merge from #779

* iterating

* mlrval/format.go

* mlrval/copy.go

* bifs/arithmetic_test.go

* iterate on bifs/collections_test.go

* mlrval_cmp.go

* mlrval JSON iterate

* iterate applying mlrval refactors to dependent packages

* first clean compile in a long while on this branch

* results of first post-compile profiling

* testing

* bugfix in ofmt formatting

* bugfix in octal-supporess

* go fmt

* neaten

* regression tests all passing
2021-12-20 23:56:04 -05:00