miller/c/todo.txt

================================================================
BUGFIXES

:D

================================================================
NON-RELEASE:

! rh/fedora/centos
* call for use-cases? (go first.)
* go through lmag article & clarify all unclears
* windows port?
? msort.o wt?

* benchmark lua per se
* benchmark lua-jit

================================================================
OTHER:

* mlr paste (sjackman)

* b.tsv bug -- to relnotes
* don't dump full help on unrecognized flag. WAY too much info. just print short, w/ info on how to get long.
* typeof -> include 'map' and 'srec'
* dump @records after @records[NR]=$*: json "1" rather than 1 indices are b04k3d
* rename dsls/ to parsers; mapping -> mappers & dsl ?
! 'Error code 1' -> clearer on regdiff fail

----------------------------------------------------------------
MAPVAR CHECKLIST:

* debt-reduction
  ! valgrinds
  ! vg.out.000 @neatener !
  - xxxes
  - prune unused functions
  ! get rid of ALL callsites where i return mv/mxv by struct-value but do not copy string-pointers
  - local-stack renames
  - mlhmmv renames
  - mlhmmv.h reorg
  - map-in vs. map-out ... xval ... mxv_t ... ?
* func/subr mapvar args; func mapvar retval
* emit-lashed debug
* apply mapvar contexts to some built-ins: length, typeof, ... ?
* allow empty maps/levels in oosvars as well as map-literals
* print mapvar; dump scalarvar

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
things which are always maps:
* maplit
* full srec
* full oosvar

things which can be maps:
* oosvar
* localvar
* func retval
* func args

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
! func retval typedecl: 'f(int a): map {...}' ?
  -> typedecls for TYPE_MASK_INT|TYPE_MASK_ABSENT ? e.g. '?int' or some such?

w lashed -- code-renames/debt-reduction/valgrinds first, before digging further

! reg_test/run to-do items

mlr_dsl_cst_output_statements.c:
k dump w/ no args
k dump maplit -- ensure scalar-out is printed
k dump full-srec -- ensure scalar-out is printed
k dump full-oosvar -- ensure scalar-out is printed
k dump oosvar -- ensure scalar-out is printed
k dump localvar -- ensure scalar-out is printed
- dump func retval -- ensure scalar-out is printed

k emit full oosvar
k emit oosvar
k emit localvar -- check lashed; UTs
k emit mapliteral -- check lashed; UTs
- emit func retval

- emit-lashed full oosvar
k emit-lashed oosvar
k emit-lashed localvar -- check lashed; UTs
k emit-lashed mapliteral -- check lashed; UTs
- emit-lashed func retval

mlr_dsl_cst_for_map_statements.c:
k for-loop over maplit
k for-loop over localvar
- for-loop over func retval

function_manager.c:
? - parameter marshaling in general rval context

mlr_dsl_cst_assignment_statements.c:
- parameter marshaling in single-rhs-assign-to-map-lhs context
.
k assign LHS full-srec          ? assign RHS maplit
k assign LHS full-srec          ? assign RHS full-srec
k assign LHS full-srec          ? assign RHS full-oosvar
k assign LHS full-srec          ? assign RHS oosvar
k assign LHS full-srec          ? assign RHS localvar
- assign LHS full-srec          ? assign RHS func-retval
.
- assign LHS full-oosvar        ? assign RHS maplit
- assign LHS full-oosvar        ? assign RHS full-srec
- assign LHS full-oosvar        ? assign RHS full-oosvar
- assign LHS full-oosvar        ? assign RHS oosvar
- assign LHS full-oosvar        ? assign RHS localvar
- assign LHS full-oosvar        ? assign RHS func-retval
.
k assign LHS oosvar             ? assign RHS maplit
k assign LHS oosvar             ? assign RHS full-srec
k assign LHS oosvar             ? assign RHS full-oosvar
k assign LHS oosvar             ? assign RHS oosvar
k assign LHS oosvar             ? assign RHS localvar
- assign LHS oosvar             ? assign RHS func-retval
.
k define LHS map-typed-localvar ? assign RHS maplit
k define LHS map-typed-localvar ? assign RHS full-srec
k define LHS map-typed-localvar ? assign RHS full-oosvar
k define LHS map-typed-localvar ? assign RHS oosvar
k define LHS map-typed-localvar ? assign RHS localvar
- define LHS map-typed-localvar ? assign RHS func-retval
.
k define LHS untyped-localvar   ? assign RHS maplit
k define LHS untyped-localvar   ? assign RHS full-srec
k define LHS untyped-localvar   ? assign RHS full-oosvar
k define LHS untyped-localvar   ? assign RHS oosvar
k define LHS untyped-localvar   ? assign RHS localvar
- define LHS untyped-localvar   ? assign RHS func-retval
.
k assign LHS localvar           ? assign RHS maplit
k assign LHS localvar           ? assign RHS full-srec
k assign LHS localvar           ? assign RHS full-oosvar
k assign LHS localvar           ? assign RHS oosvar
k assign LHS localvar           ? assign RHS localvar
- assign LHS localvar           ? assign RHS func-retval

* typedecl annotation for func retvals: 'func f(): bool'

----------------------------------------------------------------
* 'not yet' UT cases

----------------------------------------------------------------
! we are going to need builtin functions which deal with maps.
  - length(@map) and length(map), proxy for NF. make it work with length({...}) and length($*).
  - typeof(@map) and typeof(map) = "map"; typeof($*) = "srec".
  - depth on maps
  - deepcount on maps
  - split/join: but these are on lists. and would benefit from infix operators?
  - variadic list-cat function ...
  - variadic map-cat function ...
  - ?

----------------------------------------------------------------

* steps:

  ! parameter marshal
    - all UDFs need multiplexed marshaling: compound vs. scalar types
    - so all (even internally defined) functions do
    - at internally defined functions, check is_terminal

    > $x = f(a, b)
    > $x = f($a, b)

  ! @v=@* is a syntax error

  - LHS define/assign special cases:

    > assign local from xxx:
      a = {...}
      a = $*
      a = @*
      a = @b
      a = @b[c]
      a = b
      a = b[c]
      a = f(...)
      a[b] = f(...)

    > assign srec from local or function call:
      $* = {...}
      $* = a
      $* = f(...)
    > assign full oosvar from local or function call:
      @* = {...}
      @* = a
      @* = f(...)
    > assign oosvar from local or function call:
      @b = {...}
      @b = a
      @b[c] = a
      @b = f(...)
      @b[c] = f(...)

  k all other (non-special-case) LHS situations
    > coded; just need more UT coverage.

  k RHS expr eval: terminality as already for oosvars
    > coded; just need more UT coverage.

  k for-local -- with keylist etc. just as for oosvars
    > coded; just need more UT coverage.

* needs assert_scalar in userspace

* iterating:
  k save off a copy of the mlr exe before this for timings later
  k put mv_t -> mlhmmv_value_t @ localstack etc. but keep using u.mlrval and assume u.is_terminal
  ~ add in the grammar mods for each case one at a time & stub out the alloc/handle sites one at a time
  ~ fullimpl the alloc/handle sites one at a time
  ! timings before & after ! really i don't want to ship without the map feature but its cost
    should be known.
  - doc re what you can do: @-vars and locals are scalars except for the enumerated contexts

* type-decl assert RHS for map/non-map reads, and doc that:
  this should throw an error:
    mlr --from s put 'int a=1;var b=a[2]'

* type-decl assert:
  map a;
  a[1][2] = 3;
  map b = a;
  map c = a[1];
  int d = a[1][2];

================================================================
FUNCTIONALITY FOR 5.0.0:

* cat -n -g

* mapvar iterate

* map literals
  - allow assignment to oosvar or local or srec
  - requires support for empty maps/levels
  - remove the emptystrip feature from oosvars? update the doc if so. also, another backwards-incompatible change.
  ! allow $* @* @x varx all in map-literals in value nodes, with copy
  - present at AST; needs CST work

? map-literal grammar as separate parser?
  - replace existing json parser?
  - use this for streaming parser?
  - implement byte-getter with fgetc to keep reading until closing bracket?

? 'null' JSON syntax -- what syntactic meaning for miller? mv_absent or mv_empty, neither would render back as 'null'
  on output. yet i also don't want another null-type.

* support JSON arrays:
  - allow them in map-literal input
  - allow them in JSON record input
  - add mlhmmv array type, complementing map type
  - negative-index read semantics (backwards index)
  - out-of-bounds read semantics (mv_absent)
  - -out-of-bounds write semantics (array extend)
  - for srec-to-mlhmmv: loop over level & if all integer keys & all in sequence, treat as array

* typedecl:
  - eliminate 'map a' and replace with 'map a = {}'

* array/list types? [a,b,c]
* set types? {a,b,c} (vs. map {a:1,b:2,c:3})
* maybe overkill (map covers array & set) but with those there would be solid collections coverage.

----------------------------------------------------------------
EXECUTION TRACE:

* work through the grammar & neaten up pnode->text instances, now that these names are user-visible.
* mld. note it's an AST dump, not source-level.
* maybe go through and replace (...)-style with AST-to-source formatting ...
* UT
! this is all very cute tracing at source-level but would be significantly more useful with values shown.

----------------------------------------------------------------
SYNTAX ERROR IMPROVEMENT:

* try to get some source-level info in there.
  http://archive.oreilly.com/pub/a/linux/excerpts/9780596155971/error-reporting-recovery.html

================================================================
UT FOR 5.0.0:

* all assert_X() cases
* all type-decl cases: local/forbind/argpass; initial assign & reuse thereafter.

* UT prog w/ LOCAL_STACK_TRACE & LOCAL_STACK_BOUNDS_CHECK enabled
* env lhs expr
* dump locals & named/subscripted oosvars
* emit locals & named/subscripted oosvars
* for-records-counts example

================================================================
HYGIENE FOR 5.0.0:

* leading-whitespace cleanup
* xxxes
* valgrinds
* u. x 2 from union to struct & retest

================================================================
DOC FOR 5.0.0:

TYPE-CHECKING:
* assert_X() functions
* doc unset of localvars along w/ unset elsewheredoc

OOSVAR SINGLE FOR:
* oosvar-single-for mlh
* oosvar-single-for mld
* find & update cookbook/ref/whateveritwas ...

COOK:
* n-grams ... post these once mapvars are in place.
* mlr --from ~/misc/words --nidx filter 'n=strlen($1);4<=n&&n<=8' then sample -k 10
* use: https://github.com/petewarden/dstkdata
* for-records-counts example
* ./se/ examples

DEBUGGING:
* disclaimer about the Miller DSL having the usual plusses and minuses of a dynamically typed language:
  rapid prototyping and rapid access to data -- e.g. '$y=$x**2' should be free of syntactic bric-a-brac
  & the miller dsl is a place to get just that. but type-mixing can be unintentional (things really are
  implcitly typed in our heads).
* additional disclaimer: type-heterogeneity is a feature, e.g. 'for (k, v in $*) { ... }'
* absent-null & skipped assigns are a feature but aren't typo-proof:
  o typeof()
  o print/eprint
  o isX()
  o assertX() (once I write them)
  o type decls (once I write them)
* 'syntax error' is cryptic
  o ; is a common omission (separator not terminator)
  o no ++ or -- (not C-like)
  o curly braces are required in if/for/while even w/ single-statement bodies (not C-like)
* execution-trace option: disclaim it's an AST dump, not source-level.
* dump locals & named/subscripted oosvars
* emit locals & named/subscripted oosvars

PERF:
* mand bench somewhere; other benches.
* mkv2/3/4/5 somewhere; also cookbook. (better w/ mappable vars ... maybe for 5.1.0)
* xsv notes; update rust/go experiment info

-f/-e:
* emph it's just concat so watch for semicolons
* initial -e '@a=1' w/ -f '@a = ispresent(@a) ? @a : 0' as argv equiv

KW autodoc:
* kw autodoc mods for func/subr/call/return; also add NR PI E etc. to keyword help.

  E ENV FILENAME FILENUM FNR NF NR PI

To file:
* note mlr -l -L -f -F -k -K rather prominently at mlh.
* mlh/mld func/subr @ filter
* absent vs. empty; unset $x vs. $x="". no null keyword.
* env lhs expr

filter expansion:
* mlh
* mld main sections; also search around elsewhere

vars:
* clarify absent-null for variable assignments esp local. xref to homogeneity feature.

covers:
* func-def example @ cover x n
* triple-for example @ cover x n

@ new print/printn section:
* print/printn zary

mlh:
* func-def example @ mlr put -h ?
* triple-for example @ mlr put -h ?

func/subr:
* no begin/end/func/subr within func
* (lhs) func no subr call
* locals in all curly-braced control structures; top-level locals; scope, extent, freeing.
* no begin/end/func/subr within subr
* note subrs have a different namespace (log example); each kind may not be multiply defined.
* local @ RHS before @ LHS: absent skips assign
* a=2 before local a=1 -- why to even use local (when optional & when necessary).
* cookbook examples

file-formats page:
* --c2p, --c2j etc ...
* --pprint --barred; emph for output only

perf note:
* C stuff performant w/r/t other compiled langs (put up getlines samples)
* DSL stuff performant w/r/t ruby/python (mand bench & maybe others)

================================================================
5.0.0 RELEASE NOTES:

MAJOR BECAUSE:
* new keywords
* no type-inference on literals: "3"+4 != 7
* note this isn't huge. as w/ previous majors.

DSL:
* func & subr. acknowledge awk yet again.
* triple-fors
* multiple filter/put -f with examples @ new mld section.
* variadic min & max
* unset for locals
* substr func
* ENV @ LHS (putenv), and why (popens)
* # in the lexer so #-strings are OK now (show an example)

VERBS:
* most-frequent/least-frequent
* seqgen

I/O:
* bare print/printn
* --c2p et al.
* --pprint --barred

DOC:
* 10-min
* inverse-transform-sampling example at cookbook.
* http://johnkerl.org/miller/doc/cookbook.html#Memoization_with_out-of-stream_variables
* mlr -F and -K. Note -l -L -f -F -k -K.
* go/d/rust cmps @ getline
* ruby/python cmps @ mand

* final xxx check
* final valgrind check

================================================================
5.1.0: TRACE // SYNTAXERROR // **ALLRECAGG** // FCNALS

? maybe some sort of collections logic to complement for-loops? important, or just fun to code up?

----------------------------------------------------------------
5.1.0 ideas:

!!! multi-field x many verbs
  ! -f/-r field-name-spec opportunities throughout
    - bar
    - count-distinct
    - histogram
    - label
    - merge-fields -x
    - nest
    - reorder
    - sec2gmt/seg2gmtdate
    - sort
    * stats1
    * stats2
    - step
    - top
    ? join
    ? decimate -g
    ? head -g
    ? sample -g
    ? tail -g
    - uniq -g

! more than 'syntax error' from lemon-parse failures ... lemon API research

? some sort of debug/single-step/trace ... include the AST in the CST & reverse-generate ?

? formalize _ for skipvars?

? strict mode:
  - absent?
  - local shadow for-loop boundvars / func/subr args?

* seqgen as input lrec "reader"? the seqgen verb doesn't (and shouldn't) step NR.

* emph override wording: from rel notes.
  "Likewise --ofs, --ors, --ops, --jvstack, and all other output-formatting
  options from the main help at mlr -h and/or man mlr default to the main
  command-line options, and may be overridden with flags supplied to mlr put and
  mlr tee."

? some help for keyword clashes? e.g. 'func E(x) { ... }' ?
? multiple-valued function returns?? map-returns?

* bargv0 throughout, replacing "mlr" and argv[0] both. (former against name-changes; latter against /usr/bin/mlr.)

? mlr step -a shift --by {n}
? --imd ?

----------------------------------------------------------------
? double-quote-aware (non-lite) nidx & dkvp options?
? comma-number -- using locale?

? triple-for with boundvars?
? user-defined functions?
? localvars w/ decl syntax?

! multiple -f w/ append so linker is cat
* MLRPATH

----------------------------------------------------------------
? narrative/slides doc ...
? stdin-then-tail-f

* find a more elegant way to do no-semicolon-after-block but don't block 4.1.0
? autodetect format by file extension -- ?
? c-ify the rectangularize feature? unset fields; and?or? emit fields; and?or?
* two remaining valgrinds -- both venial

? CONVFMT? IGNORECASE? MLRPATH?

* lrec get followed by put/remove: getext variant returning node for unlink, valpoke, or null == append to avoid
  double-searching.
* 64-bit lengths for containers. test with 5-billion-integer-seq data.

----------------------------------------------------------------
MORE

* look into 'perf'
* linreg in terms of stats only ... check py book
* cat/cut langcomps (w/ gh links) -> perf page

* pipe-viewer-like feature to stderr?

* stats1/stats2 sliding-window feature? and/or with ewma-coefficients (much easier)
  - mean/stddev/var; skew/kurt?
  - linregs; corr/cov?
  ? also, option of weighted stats w/ explicit weights field?
  ? maybe just EWMA with well-known sumw followed by then-chaining. write up the weights if so?

* tbin/ok -> cookbook
* debian screenshot
* ruby @ optextdep @ mld; poki+mkman
* stdin filename keyword for read-from-file-then-tail-f mode (e.g. mlr etc)
  - needs refactor for lrec_reader_alloc callsite
* perf page: (1) redo; (2) note GNU/etc; (3) compare to mawk (http://invisible-island.net/mawk/)
* EOS comments thruout
* valgrind note @ new dev page/section
* join: final sllv_free in destructor (lo-pri)
* anim ref https://github.com/edi9999/path-extractor

----------------------------------------------------------------
COOKBOOK/FAQ/ETC.:

* cookbook:
  - eval stuff from https://github.com/johnkerl/miller/issues/88

    $ mlr --csvlite stats2 -a linreg-pca  -f x,y x
    x_y_pca_m,x_y_pca_b,x_y_pca_n,x_y_pca_quality
    1.030300,0.949250,4,0.999859
    $ mlr --csvlite --odkvp --ofs semicolon stats2 -a linreg-pca  -f x,y x
    x_y_pca_m=1.030300;x_y_pca_b=0.949250;x_y_pca_n=4;x_y_pca_quality=0.999859
    $ eval $(mlr --csvlite --odkvp --ofs semicolon stats2 -a linreg-pca  -f x,y x)
    $ echo $x_y_pca_m
    1.030300

  - hold-and-fit regressor doc: 'then put' for residuals; note avoids two-pass &
    the saving of fit parameters
  - histo w/ min/max is effectively 2-pass (unless you have prior knowledge about the data).
    note count-distinct w/ int() func.
  - two-pass lin/logi reg vs. hold-and-fit.

  - very specific R/mysql/etc inouts

  - polyglottal dkvp/etc production.
  - very specific examples of sed/grep/etc preprocessing to structurize semi-structured data (e.g. logs)

  - checku.dash -> cookbook

* faqents:
  - rsum as proxy for per-record/agg-only mixed output

* other doc besides cookbook & faq:
  - R doc:
    ! xref @ covers x 2
    ! be very clear streaming vs. dataframe -- each has things the other can't do
    ! emph mlr has light stats but for heavyweight analysis use R et al.

* --mmap @ mlr -h

================================================================
* ect feature?
  -> maybe better in cookbook ...
  - in1 optional: t (epoch seconds); default systime()
  - in2: nleft
  - in3 optional: target #/field name
  - in optional: -s flag or not
  - out1: etchours
  - out2: etcstamp

  o expose mapper_stats2_alloc
  o expose mapper_cut_alloc
  o encapsulate the following:
    mlr put '$t=systime()' \
      then filter 'NR>4' \
      then  put '$nleft=$target-$n' \
      then stats2 -s -a linreg-pca -f t,nleft \
      then put '$etc= -$t_n_pca_b/$t_n_pca_m; $etcstamp=sec2gmt($etc); $etchours=($etc-systime())/3600.0'

* mlr step -a from-first -f t \
    then cut -o -f t_from_first,ntodo \
    then step -a ewma -d 0.005,0.01,0.1 -o a,b,c -f ntodo \
    then stats2 -s -a linreg-pca -f \
      t_from_first,ntodo,t_from_first,ntodo_ewma_a,t_from_first,ntodo_ewma_b,t_from_first,ntodo_ewma_c \
    then put '
      $ect0 = -$t_from_first_ntodo_pca_b/$t_from_first_ntodo_pca_m;
      $ecta = -$t_from_first_ntodo_ewma_a_pca_b/$t_from_first_ntodo_ewma_a_pca_m;
      $ectb = -$t_from_first_ntodo_ewma_b_pca_b/$t_from_first_ntodo_ewma_b_pca_m;
      $ectc = -$t_from_first_ntodo_ewma_c_pca_b/$t_from_first_ntodo_ewma_c_pca_m
    ' \
    then cut -o -f t_from_first,ect0,ecta,ectb,ectc

----------------------------------------------------------------
* introduce a fourth, padding separator for all formats? (for leading/trailing strip/skip.)
  o allows 'x = 10' in DKVP
  o allows right-justified keys in XTAB

* hold-and-emit fraction?

* statsn covar, ols, logistic: port material from my stats_m/sackmat_m for much of that

* uni/multivariate logistic for ternary & above?

? wiki quickselect ?

* hash-collision ifdef instrumentation -> maybe find a better hash function out there
* lemon in-dir -- cf wiz note
* gprof link with -lc on FreeBSD -- ?

================================================================
UT/REG
* ut cat/X/cat for all X
* ut tac/X/cat for all X
* ut cat/X/tac for all X
* ut tac/X/tac for all X
* ut multi-csv I/O: include --icsvlite --odkvp and --idkvp --ocsv, as well as --csv cases
* ut het-xtab out
* ut modulus operator
* ut make should-fail machinery & use it for null-key dkvp cases.
* ut all mathlib funcs
* ut int/float/string
* ut roundm
* ut join with left/right-prefix

================================================================
DOC

* Note that PCA is better than OLS for roundoff error (sum of squares ...):
  grep red data/multicountdown.txt | head -n 13 | mlr --opprint stats2 -a linreg-ols -f t,count
  grep red data/multicountdown.txt | head -n 14 | mlr --opprint stats2 -a linreg-ols -f t,count

================================================================
IMPROVEMENTS

* run go/d/etc on sprax & include #'s in perf pg, and/or rm xref in the latter & just post xlang perf #'s there
* link to gh/jk/m xlang impls ... and/or cardify their sources :) ... or maybe just link to gh/jk/m xlang dir
* ack c impl has been repeatedly optimized but even the original version (also cutc.c ...) outperforms

* update t1.rb including numeric sort; fix appropriateness of -t=

* more use of restrict pointers ... ?

================================================================
PYTHON
* pgr + stats_m same I/O modules??

================================================================
FYI

shell: mlr put '$z=$x+$y'
lldb:  run put "\$z=\$x+\$y"

http://include-what-you-use.org/

----------------------------------------------------------------
https://fedoraproject.org/wiki/How_to_create_an_RPM_package
https://wiki.centos.org/HowTos/Packages/ContributeYourRPMs
https://lists.centos.org/pipermail/centos/2012-September/129227.html
https://fedoraproject.org/wiki/Join_the_package_collection_maintainers
https://fedoraproject.org/wiki/How_to_get_sponsored_into_the_packager_group
https://fedoraproject.org/wiki/Package_Review_Process
https://docs.fedoraproject.org/ro/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch11s03.html
http://wiki.networksecuritytoolkit.org/nstwiki/index.php/HowTo_Create_A_Patch_File_For_A_RPM

================================================================
git remote add upstream https://github.com/Homebrew/homebrew
git fetch upstream
git rebase upstream/master
git checkout -b miller-3.4.0
shasum -a 256 ../mlr-3.4.0.tar.gz
edit Library/Formula/miller.rb
git add ...
git commit -m 'miller 3.4.0'
git push -u origin miller-3.4.0
submit the pull request

----------------------------------------------------------------
Squash commits by:
  brew update
  git checkout $YOUR_BRANCH
  git rebase --interactive origin/master
  mark each commit other than the first as "squash" or "fixup"
  git push -f

http://codeinthehole.com/writing/pull-requests-and-other-good-practices-for-teams-using-github/