mirror of
https://github.com/johnkerl/miller.git
synced 2026-08-01 12:11:15 +00:00
neaten
This commit is contained in:
John Kerl
parent
efeac7f36d
commit
2ba4a364ff
18 changed files with 160 additions and 90 deletions
78
go/README.md
78
go/README.md
|
|
@ -1,13 +1,75 @@
|
|||
# Status of the Go port
|
||||
|
||||
* This is not necessarily a full Go port of Miller. At the moment, it's a little spot for some experimentation. Things are very rough and very iterative and very incomplete.
|
||||
* This is not necessarily a full Go port of Miller. At the moment, it's a little spot for some experimentation. Things are very rough and very iterative and very incomplete. I don't commit to finishing a Go port but I very much hope to.
|
||||
* One reason Miller exists is to be a useful tool for myself and others; another is it's fun to write. At bare minimum, I'll re-teach myself some Go.
|
||||
* In all likelihood though this will turn into a full port which will someday become Miller 6.0.
|
||||
* In all likelihood, though, this will turn into a full port which will someday become Miller 6.0.
|
||||
* I hope to retain backward compatibility at the command-line level as much as possible.
|
||||
* Benefits:
|
||||
* The lack of a streaming (record-by-record) JSON reader in the C implementation (https://github.com/johnkerl/miller/issues/99) is immediately solved in the Go implementation.
|
||||
* The quoted-DKVP feature from https://github.com/johnkerl/miller/issues/266 will be easily addressed.
|
||||
* String/number-formatting issues in https://github.com/johnkerl/miller/issues/211 https://github.com/johnkerl/miller/issues/178 https://github.com/johnkerl/miller/issues/151 https://github.com/johnkerl/miller/issues/259 will be fixed during the Go port.
|
||||
* I think some DST/timezone issues such as https://github.com/johnkerl/miller/issues/359 will be easier to fix using the Go datetime library than using the C datetime library
|
||||
* Benefits of porting to Go:
|
||||
* The lack of a streaming (record-by-record) JSON reader in the C implementation ([issue 99](https://github.com/johnkerl/miller/issues/99)) is immediately solved in the Go implementation.
|
||||
* The quoted-DKVP feature from [issue 266](https://github.com/johnkerl/miller/issues/266) will be easily addressed.
|
||||
* String/number-formatting issues in [issue 211](https://github.com/johnkerl/miller/issues/211), [issue 178](https://github.com/johnkerl/miller/issues/178), [issue 151](https://github.com/johnkerl/miller/issues/151), and [issue 259](https://github.com/johnkerl/miller/issues/259) will be fixed during the Go port.
|
||||
* I think some DST/timezone issues such as [issue 359](https://github.com/johnkerl/miller/issues/359) will be easier to fix using the Go datetime library than using the C datetime library
|
||||
* The code will be easier to read and, I hope, easier for others to contribute to.
|
||||
* In the meantime I will still keep fixing bugs, doing some features, etc in C on Miller 5.x.
|
||||
* In the meantime I will still keep fixing bugs, doing some features, etc. in C on Miller 5.x -- in the near term, support for Miller's C implementation continues as before.
|
||||
|
||||
# Directory structure
|
||||
|
||||
Miller is a multi-format record-stream processor, where a *record* is a
|
||||
sequence of key-value pairs. The basic *stream* operation is:
|
||||
|
||||
* *read* records in some specified file format;
|
||||
* *map* the input records to output records in some user-specified way, using a *chain* of *verbs* (sort, filter, cut, put, etc.);
|
||||
* *write* the records in some specified file format.
|
||||
|
||||
## Directory-structure overview
|
||||
|
||||
So, in broad overview, the key packages are:
|
||||
|
||||
```
|
||||
src/miller/stream -- connect input -> mapping -> output via Go channels
|
||||
src/miller/input -- read input records
|
||||
src/miller/mapping -- map input records to output records
|
||||
src/miller/output -- write output records
|
||||
```
|
||||
|
||||
## Directory-structure details
|
||||
|
||||
### Dependencies
|
||||
|
||||
* Miller dependencies are all in the Go standard library, except a couple local ones:
|
||||
* Insertion-ordered (order-preserving) maps from [gitlab.com/c0b/go-ordered-json](https://gitlab.com/c0b/go-ordered-json):
|
||||
* If you have a JSON data record `{"x":3,"y":4,"z":5}` then the keys `x,y,z` should stay that way. This package makes that happen.
|
||||
* GOCC lexer/parser code-generator from [github.com/goccmack/gocc](https://github.com/goccmack/gocc):
|
||||
* This package defines the grammar for Miller's domain-specific language (DSL) for the Miller `put` and `filter` verbs. And, GOCC is a joy to use. :)
|
||||
|
||||
|
||||
```
|
||||
src/localdeps/ordered
|
||||
src/github.com/goccmack
|
||||
src/miller
|
||||
```
|
||||
|
||||
### More
|
||||
|
||||
```
|
||||
mlr.go
|
||||
src/miller/lib
|
||||
src/miller/containers
|
||||
|
||||
src/miller/cli
|
||||
src/miller/clitypes
|
||||
src/miller/stream
|
||||
src/miller/input
|
||||
src/miller/mapping
|
||||
src/miller/output
|
||||
|
||||
src/miller/mappers
|
||||
src/miller/parsing
|
||||
src/miller/parsing/token
|
||||
src/miller/parsing/util
|
||||
src/miller/parsing/lexer
|
||||
src/miller/parsing/parser
|
||||
src/miller/parsing/errors
|
||||
src/miller/dsl
|
||||
src/miller/dsl/cst
|
||||
```
|
||||
|
|
|
|||
24
go/ppp
24
go/ppp
|
|
@ -1,24 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
echo; mlr put -v '$y = 1 || 2' /dev/null
|
||||
echo; mlr null put -v '$y = 1 || 2 || 3' /dev/null
|
||||
echo; mlr null put -v '$y = 1 || 2 && 3' /dev/null
|
||||
echo; mlr null put -v '$y = 1 && 2 || 3' /dev/null
|
||||
echo; mlr null put -v '$y = 1 ? 2 : 3' /dev/null
|
||||
echo; mlr null put -v '$y = $a + $b * $c' /dev/null
|
||||
echo; mlr null put -v '$y = $a * $b * $c' /dev/null
|
||||
echo; mlr null put -v '$y = $a ** $b ** $c' /dev/null
|
||||
echo; mlr null put -v '$[2] = 3' /dev/null
|
||||
echo; mlr null put -v '$[$y] = 4' /dev/null
|
||||
#echo; mlr null put -v '${1} = 4' /dev/null
|
||||
echo; mlr null put -v '$x = "abc"' /dev/null
|
||||
echo; mlr null put -v '$["abc"] = "def"' /dev/null
|
||||
echo; mlr null put -v '$[FILENAME] = FNR' /dev/null
|
||||
echo; mlr null put -v '$x = $a + $b + $c' /dev/null
|
||||
echo; mlr null put -v '$x = ($a + $b) + $c; $y = $a + ($b + $c); $z = $a + ($b)+ $c' /dev/null
|
||||
echo; mlr null put -v '$x = 2 * $a + $b . $c' /dev/null
|
||||
echo; mlr null put -v '$x = 2 * $a + ($b . $c)' /dev/null
|
||||
echo; mlr null put -v '$x = (NF + NR) * 7; $y = OFS . $y . "hello"' /dev/null
|
||||
echo; mlr null put -v '$x = 123. + 1e-2 / .2e3 + 1.e-3' /dev/null
|
||||
50
go/rrr
50
go/rrr
|
|
@ -1,50 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
echo; mlr --idkvp --opprint put '$j = $i + $i' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$j = $i + $x' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$j = $y + $x' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$j = $y + $i' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$j = $y + $y' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$j = $i+$i' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=$x*1e6' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=$x+1e6' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=$x+1' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=FILENAME' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=FILENUM' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=NF' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=NR' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=FNR' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=NR' s.dkvp s.dkvp
|
||||
echo; mlr --icsv --opprint put '$y=FNR' s.csv s.csv
|
||||
echo; mlr --idkvp --opprint put '$y=FNR+1' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=FNR+$i' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=FNR+3' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=FNR+3+$i' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=$i+$y' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=$i+$x' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=$x+$y' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=$x+$i' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=NR+$i' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=NR-$i' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=4-1' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=NR' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=$i' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100*NR-$i' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100*$i+$x' s.dkvp s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100*$i+$x' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100*$i/$x' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=NR/$i' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100/$i' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100//$i' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100//$x' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100.0//$i' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100.0 // $i' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100.0 / $i' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100.0 ' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100 ' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100.4 ' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=1.2 ' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100.0 / $i ' s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100.0 // $i ' s.dkvp
|
||||
12
go/todo.txt
12
go/todo.txt
|
|
@ -1,6 +1,14 @@
|
|||
----------------------------------------------------------------
|
||||
TOP OF LIST:
|
||||
* README.md at various levels
|
||||
* widen CLI coverage
|
||||
o --c2x et al.
|
||||
* widen DSL coverage
|
||||
o begin/end blocks
|
||||
o indirect field names, at LHS and RHS
|
||||
o hex inputs
|
||||
o bitwise/int operators
|
||||
o support the filter verb
|
||||
* widen reader coverage
|
||||
o TSV/ASV
|
||||
o XTAB
|
||||
|
|
@ -20,10 +28,8 @@ TOP OF LIST:
|
|||
o repeat
|
||||
o head/tail
|
||||
o sort
|
||||
o filter
|
||||
* implement mlrrc
|
||||
* widen DSL coverage
|
||||
o begin/end blocks
|
||||
o indirect field names, at LHS and RHS
|
||||
----------------------------------------------------------------
|
||||
|
||||
general plan:
|
||||
|
|
|
|||
2
go/u/README.md
Normal file
2
go/u/README.md
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
These are manually invoked sanity-check scripts, eventually to be replaced by
|
||||
duplicating the `../c/reg_test/run` framework.
|
||||
24
go/u/ppp
Executable file
24
go/u/ppp
Executable file
|
|
@ -0,0 +1,24 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
echo; mlr -n put -v '$y = 1 || 2'
|
||||
echo; mlr -n put -v '$y = 1 || 2 || 3'
|
||||
echo; mlr -n put -v '$y = 1 || 2 && 3'
|
||||
echo; mlr -n put -v '$y = 1 && 2 || 3'
|
||||
echo; mlr -n put -v '$y = 1 ? 2 : 3'
|
||||
echo; mlr -n put -v '$y = $a + $b * $c'
|
||||
echo; mlr -n put -v '$y = $a * $b * $c'
|
||||
echo; mlr -n put -v '$y = $a ** $b ** $c'
|
||||
echo; mlr -n put -v '$[2] = 3'
|
||||
echo; mlr -n put -v '$[$y] = 4'
|
||||
#echo; mlr -n put -v '${1} = 4'
|
||||
echo; mlr -n put -v '$x = "abc"'
|
||||
echo; mlr -n put -v '$["abc"] = "def"'
|
||||
echo; mlr -n put -v '$[FILENAME] = FNR'
|
||||
echo; mlr -n put -v '$x = $a + $b + $c'
|
||||
echo; mlr -n put -v '$x = ($a + $b) + $c; $y = $a + ($b + $c); $z = $a + ($b)+ $c'
|
||||
echo; mlr -n put -v '$x = 2 * $a + $b . $c'
|
||||
echo; mlr -n put -v '$x = 2 * $a + ($b . $c)'
|
||||
echo; mlr -n put -v '$x = (NF + NR) * 7; $y = OFS . $y . "hello"'
|
||||
echo; mlr -n put -v '$x = 123. + 1e-2 / .2e3 + 1.e-3'
|
||||
|
|
@ -25,7 +25,7 @@ for ifmt in csv dkvp nidx json; do
|
|||
$fnr=FNR;
|
||||
$filename=FILENAME;
|
||||
$filenum=FILENUM;
|
||||
' s.$ifmt t.$ifmt
|
||||
' u/s.$ifmt u/t.$ifmt
|
||||
echo
|
||||
done
|
||||
|
||||
|
|
@ -42,6 +42,6 @@ for ofmt in pprint csv dkvp nidx json; do
|
|||
$fnr=FNR;
|
||||
$filename=FILENAME;
|
||||
$filenum=FILENUM;
|
||||
' s.$ifmt t.$ifmt
|
||||
' u/s.$ifmt u/t.$ifmt
|
||||
echo
|
||||
done
|
||||
50
go/u/rrr
Executable file
50
go/u/rrr
Executable file
|
|
@ -0,0 +1,50 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
echo; mlr --idkvp --opprint put '$j=$i+$i' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$j=$i+$x' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$j=$y+$x' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$j=$y+$i' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$j=$y+$y' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$j=$i+$i' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=$x*1e6' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=$x+1e6' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=$x+1' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=FILENAME' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=FILENUM' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=NF' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=NR' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=FNR' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=NR' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --icsv --opprint put '$y=FNR' u/s.csv u/t.csv
|
||||
echo; mlr --idkvp --opprint put '$y=FNR+1' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=FNR+$i' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=FNR+3' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=FNR+3+$i' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=$i+$y' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$y=$i+$x' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=$x+$y' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=$x+$i' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=NR+$i' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=NR-$i' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=4-1' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=NR' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=$i' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100*NR-$i' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100*$i+$x' u/s.dkvp u/t.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100*$i+$x' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100*$i/$x' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=NR/$i' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100/$i' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100//$i' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100//$x' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100.0//$i' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100.0//$i' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100.0/$i' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100.0' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100.4' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=1.2' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100.0/$i' u/s.dkvp
|
||||
echo; mlr --idkvp --opprint put '$z=100.0//$i' u/s.dkvp
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
echo ================================================================
|
||||
mlr --idkvp --opprint cat then tac s.dkvp t.dkvp
|
||||
mlr --idkvp --opprint cat then tac u/s.dkvp u/t.dkvp
|
||||
echo ================================================================
|
||||
mlr --idkvp --opprint cat -n then tac s.dkvp t.dkvp
|
||||
mlr --idkvp --opprint cat -n then tac u/s.dkvp u/t.dkvp
|
||||
echo ================================================================
|
||||
mlr --idkvp --opprint cat -n then tac then cat -n s.dkvp t.dkvp
|
||||
mlr --idkvp --opprint cat -n then tac then cat -n u/s.dkvp u/t.dkvp
|
||||
echo ================================================================
|
||||
Loading…
Add table
Add a link
Reference in a new issue