================================================================
BUGFIXES

:D

================================================================
TOP-OF-LIST SUMMARY

! oosvar keyed mean?!?

! rh/fedora/centos mlr-3.5.0 or mlr-4.0.0

? call for use-cases? (go first.)
* dsls deps still not quite right?

* lrec get followed by put/remove: getext variant returning node for unlink, valpoke, or null == append to avoid
  double-searching.
* reduce parameter-marshaling by pevalbag; measure perf delta (suspect negligible)
* typed overlay -> mlhmmv rather than lhmsv
* 64-bit lengths for containers. test with 5-billion-integer-seq data.
* --strict? mlh/mld if so.

* 4.1.0:
  ? autodetect format by file extension -- ?
  ? mlr -n ?
  ? komosa multiops?
    ? for x $* { for y { @v["a"] { ... } } ?

================================================================
* relcut material since 3.5.0:
  - oosvars
  - begin/end
  - emit/all
  - unset/all
  - bare-boolean and pattern-action
  - put -q: equiv to putting 'filter false'
  - += et al. were ever announced -- fix intro release & note
  - double-backslashing bug in sub/gsub
  - --from, & why: up-arrow & append then-chain

================================================================
OOSVAR EXPERIMENTS:

* check for functionality -- this is my best chance to do a full-featuredness check for oosvars & emit
* also do timing comparisons

* list:
  - bootstrap?
  - count-distinct?
  - cut?
  - decimate? easy now with NR%10 if not -g; what about with -g?
  - group-by?
  - group-like?
  - having-fields?
  - head?
  - histogram?
  - (rename? reorder?)
  - sample, w/ & w/o -g?
  - stats1 count,sum,mean,stddev,var,skew,kurt; mode; min,max?
  - stats2 linregs,r2,corr,cov?
  - step delta,from-first,ratio,rsum,counter,ewma?
  - tail?
  > top?
  - uniq?

* perf notes:

  $ time mlr stats1 -a mean -f x ../data/big.dkvp
  x_mean=0.499567
  real    0m1.486s 0m1.500s 0m1.637s

  $ time mlr put -q 'begin @x_count=0;begin @x_sum=0.0;@x_sum = @x_sum+$x;@x_count=@x_count+1; end @x_mean=@x_sum/@x_count; end emit @x_mean'  ../data/big.dkvp
  x_mean=0.499567
  real    0m2.262s 0m2.371s 0m2.356s

  mlr put 'begin @a=0.1;begin @b=0.9;$e=NR==1?$x:@a*$x+@b*@e;@e=$e' then step -a ewma -d 0.1 -f x ../data/big.dkvp
  time mlr put 'begin @a=0.1;begin @b=0.9;$e=NR==1?$x:@a*$x+@b*@e;@e=$e' ../data/big.dkvp > /dev/null
  real  0m3.976s 0m3.971s 0m3.944s
  time mlr step -a ewma -d 0.1 -f x ../data/big.dkvp > /dev/null
  real  0m3.250s 0m3.283s 0m3.300s

*
  $ time   ../c/mlr put -q '@min=min(@min,$x);@max=max(@max,$x); end{emitf @min, @max}'  ../data/big.dkvp
  min=0.000000,max=0.999999

  real    0m3.270s
  user    0m3.196s
  sys 0m0.072s

  $ time ../c/mlr stats1 -a min,max -f x ../data/big.dkvp
  x_min=0.000000,x_max=0.999999

  real    0m1.805s
  user    0m1.736s
  sys 0m0.062s

  $ mlr --opprint step -a delta -f x ../data/small
  a   b   i  x                   y                   x_delta
  pan pan 1  0.3467901443380824  0.7268028627434533  0
  eks pan 2  0.7586799647899636  0.5221511083334797  0.411890
  wye wye 3  0.20460330576630303 0.33831852551664776 -0.554077
  eks wye 4  0.38139939387114097 0.13418874328430463 0.176796
  wye pan 5  0.5732889198020006  0.8636244699032729  0.191890
  zee pan 6  0.5271261600918548  0.49322128674835697 -0.046163
  eks zee 7  0.6117840605678454  0.1878849191181694  0.084658
  zee wye 8  0.5985540091064224  0.976181385699006   -0.013230
  hat wye 9  0.03144187646093577 0.7495507603507059  -0.567112
  pan wye 10 0.5026260055412137  0.9526183602969864  0.471184

  $ mlr --opprint put '$x_delta=ispresent(@last)?$x-@last:0;@last=$x' ../data/small
  a   b   i  x                   y                   x_delta
  pan pan 1  0.3467901443380824  0.7268028627434533  0
  eks pan 2  0.7586799647899636  0.5221511083334797  0.411890
  wye wye 3  0.20460330576630303 0.33831852551664776 -0.554077
  eks wye 4  0.38139939387114097 0.13418874328430463 0.176796
  wye pan 5  0.5732889198020006  0.8636244699032729  0.191890
  zee pan 6  0.5271261600918548  0.49322128674835697 -0.046163
  eks zee 7  0.6117840605678454  0.1878849191181694  0.084658
  zee wye 8  0.5985540091064224  0.976181385699006   -0.013230
  hat wye 9  0.03144187646093577 0.7495507603507059  -0.567112
  pan wye 10 0.5026260055412137  0.9526183602969864  0.471184

  $ mlr --opprint step -a delta -f x -g a ../data/small
  a   b   i  x                   y                   x_delta
  pan pan 1  0.3467901443380824  0.7268028627434533  0
  eks pan 2  0.7586799647899636  0.5221511083334797  0
  wye wye 3  0.20460330576630303 0.33831852551664776 0
  eks wye 4  0.38139939387114097 0.13418874328430463 -0.377281
  wye pan 5  0.5732889198020006  0.8636244699032729  0.368686
  zee pan 6  0.5271261600918548  0.49322128674835697 0
  eks zee 7  0.6117840605678454  0.1878849191181694  0.230385
  zee wye 8  0.5985540091064224  0.976181385699006   0.071428
  hat wye 9  0.03144187646093577 0.7495507603507059  0
  pan wye 10 0.5026260055412137  0.9526183602969864  0.155836

  $ mlr --opprint put '$x_delta=ispresent(@last[$a])?$x-@last[$a]:0;@last[$a]=$x' ../data/small
  a   b   i  x                   y                   x_delta
  pan pan 1  0.3467901443380824  0.7268028627434533  0
  eks pan 2  0.7586799647899636  0.5221511083334797  0
  wye wye 3  0.20460330576630303 0.33831852551664776 0
  eks wye 4  0.38139939387114097 0.13418874328430463 -0.377281
  wye pan 5  0.5732889198020006  0.8636244699032729  0.368686
  zee pan 6  0.5271261600918548  0.49322128674835697 0
  eks zee 7  0.6117840605678454  0.1878849191181694  0.230385
  zee wye 8  0.5985540091064224  0.976181385699006   0.071428
  hat wye 9  0.03144187646093577 0.7495507603507059  0
  pan wye 10 0.5026260055412137  0.9526183602969864  0.155836

  time mlr --opprint step -a delta -f x then check ../data/big.dkvp
    real 0m2.794s
    user 0m2.731s
    sys  0m0.056s

  time mlr --opprint put '$x_delta=ispresent(@last)?$x-@last:0;@last=$x' then check ../data/big.dkvp
    real 0m4.037s
    user 0m3.973s
    sys  0m0.057s

  time mlr --opprint step -a delta -f x -g a then check ../data/big.dkvp
    real 0m3.016s
    user 0m2.950s
    sys  0m0.059s

  time mlr --opprint put '$x_delta=ispresent(@last[$a])?$x-@last[$a]:0;@last[$a]=$x' then check ../data/big.dkvp
    real 0m5.792s
    user 0m5.715s
    sys  0m0.069s

*
  $ cat f
  @n += 1;
  @sumx += $x;
  @sumx2 += $x**2;
  end{
    @mean = @sumx / @n;
    @var = (@sumx2 - @mean * (2 * @sumx - @n * @mean)) / (@n - 1);
    @stddev = sqrt(@var);
    emit all
  }

  $ time mlr put -q -f g ../data/big.dkvp
  n=1000000
  sumx=499567.231705
  sumx2=333050.794634
  mean=0.499567
  numer=83483.375641
  denom=999999
  var=0.083483
  stddev=0.288935

  real    0m3.869s
  user    0m3.790s
  sys 0m0.073s


  $ time mlr --oxtab stats1 -a count,sum,mean,var,stddev -f x ../data/big.dkvp
  x_count  1000000
  x_sum    499567.231705
  x_mean   0.499567
  x_var    0.083483
  x_stddev 0.288935

  real    0m2.173s
  user    0m2.111s
  sys 0m0.060s

================================================================
NON-PRE4
================================================================

* look into 'perf'
* linreg in terms of stats only ... check py book
* option for: print to stderr NR if NR%N==0 for user-specified N
* cat/cut langcomps (w/ gh links) -> perf page

* pipe-viewer-like feature to stderr?

* interpolated percentiles

? DSL comments: replace with spaces from # to EOL/EOS, whichever comes first
* stats1/stats2 sliding-window feature? and/or with ewma-coefficients (much easier)
  - mean/stddev/var; skew/kurt?
  - linregs; corr/cov?
  ? also, option of weighted stats w/ explicit weights field?
  ? maybe just EWMA with well-known sumw followed by then-chaining. write up the weights if so?

* mld re cross-record stuff is limited to stats1/2 and step
  o this will change with begin/end and oosvars
* tbin/ok -> cookbook
* debian screenshot
* lrec_evaluators cleanup re strict, redundant statements, etc.
* ruby @ optextdep @ mld; poki+mkman
* comma-number -- using locale?
* stdin filename keyword for read-from-file-then-tail-f mode (e.g. mlr etc)
  - needs refactor for lrec_reader_alloc callsite
* perf page: (1) redo; (2) note GNU/etc; (3) compare to mawk (http://invisible-island.net/mawk/)
* EOS comments thruout
* valgrind note @ new dev page/section
* join: final sllv_free in destructor (lo-pri)
* anim ref https://github.com/edi9999/path-extractor

? json arrays -> nested w/ some delimiter? only if all array elements are terminals.

* flight misc: .screenrc -> dotfiles; more dotfile currency
* cump -> one-offs

----------------------------------------------------------------
NARRATIVE INTRO:
* sql example
* logging example
* csv example
* "what do these have in common?"

----------------------------------------------------------------
COOKBOOK/FAQ/ETC.:

* cookbook:
  - eval stuff from https://github.com/johnkerl/miller/issues/88

    $ mlr --csvlite stats2 -a linreg-pca  -f x,y x
    x_y_pca_m,x_y_pca_b,x_y_pca_n,x_y_pca_quality
    1.030300,0.949250,4,0.999859
    $ mlr --csvlite --odkvp --ofs semicolon stats2 -a linreg-pca  -f x,y x
    x_y_pca_m=1.030300;x_y_pca_b=0.949250;x_y_pca_n=4;x_y_pca_quality=0.999859
    $ eval $(mlr --csvlite --odkvp --ofs semicolon stats2 -a linreg-pca  -f x,y x)
    $ echo $x_y_pca_m
    1.030300

  - hold-and-fit regressor doc: 'then put' for residuals; note avoids two-pass &
    the saving of fit parameters
  - histo w/ min/max is effectively 2-pass (unless you have prior knowledge about the data).
    note count-distinct w/ int() func.
  - two-pass lin/logi reg vs. hold-and-fit.

  - very specific R/mysql/etc inouts

  - polyglottal dkvp/etc production.
  - very specific examples of sed/grep/etc preprocessing to structurize semi-structured data (e.g. logs)

  - checku.dash -> cookbook

* faqents:
  - rsum as proxy for per-record/agg-only mixed output

* other doc besides cookbook & faq:
  - R doc:
    ! xref @ covers x 2
    ! be very clear streaming vs. dataframe -- each has things the other can't do
    ! emph mlr has light stats but for heavyweight analysis use R et al.

* --mmap @ mlr -h
* bus-insurance dev page

================================================================
* ect feature?
  -> maybe better in cookbook ...
  - in1 optional: t (epoch seconds); default systime()
  - in2: nleft
  - in3 optional: target #/field name
  - in optional: -s flag or not
  - out1: etchours
  - out2: etcstamp

  o expose mapper_stats2_alloc
  o expose mapper_cut_alloc
  o encapsulate the following:
    mlr put '$t=systime()' \
      then filter 'NR>4' \
      then  put '$nleft=$target-$n' \
      then stats2 -s -a linreg-pca -f t,nleft \
      then put '$etc= -$t_n_pca_b/$t_n_pca_m; $etcstamp=sec2gmt($etc); $etchours=($etc-systime())/3600.0'

* mlr step -a from-first -f t \
    then cut -o -f t_from_first,ntodo \
    then step -a ewma -d 0.005,0.01,0.1 -o a,b,c -f ntodo \
    then stats2 -s -a linreg-pca -f \
      t_from_first,ntodo,t_from_first,ntodo_ewma_a,t_from_first,ntodo_ewma_b,t_from_first,ntodo_ewma_c \
    then put '
      $ect0 = -$t_from_first_ntodo_pca_b/$t_from_first_ntodo_pca_m;
      $ecta = -$t_from_first_ntodo_ewma_a_pca_b/$t_from_first_ntodo_ewma_a_pca_m;
      $ectb = -$t_from_first_ntodo_ewma_b_pca_b/$t_from_first_ntodo_ewma_b_pca_m;
      $ectc = -$t_from_first_ntodo_ewma_c_pca_b/$t_from_first_ntodo_ewma_c_pca_m
    ' \
    then cut -o -f t_from_first,ect0,ecta,ectb,ectc

----------------------------------------------------------------
* introduce a fourth, padding separator for all formats? (for leading/trailing strip/skip.)
  o allows 'x = 10' in DKVP
  o allows right-justified keys in XTAB

* hold-and-emit fraction?

* statsn covar, ols, logistic: port material from my stats_m/sackmat_m for much of that

* uni/multivariate logistic for ternary & above?

? wiki quickselect ?

* sllv_free option with callback for void-star-payload free; likewise other void-star-payload containers
* double-check for off-by-one buflen in cline/sline
* hash-collision ifdef instrumentation -> maybe find a better hash function out there
* pprint join?
* lemon in-dir -- cf wiz note
* gprof link with -lc on FreeBSD -- ?

================================================================
UT/REG
* ut cat/X/cat for all X
* ut tac/X/cat for all X
* ut cat/X/tac for all X
* ut tac/X/tac for all X
* ut multi-csv I/O: include --icsvlite --odkvp and --idkvp --ocsv, as well as --csv cases
* ut het-xtab out
* ut modulus operator
* ut make should-fail machinery & use it for null-key dkvp cases.
* ut all mathlib funcs
* ut int/float/string
* ut roundm
* ut join with left/right-prefix

================================================================
DOC

* Note that PCA is better than OLS for roundoff error (sum of squares ...):
  grep red data/multicountdown.txt | head -n 13 | mlr --opprint stats2 -a linreg-ols -f t,count
  grep red data/multicountdown.txt | head -n 14 | mlr --opprint stats2 -a linreg-ols -f t,count

================================================================
IMPROVEMENTS

* run go/d/etc on sprax & include #'s in perf pg, and/or rm xref in the latter & just post xlang perf #'s there
* link to gh/jk/m xlang impls ... and/or cardify their sources :) ... or maybe just link to gh/jk/m xlang dir
* ack c impl has been repeatedly optimized but even the original version (also cutc.c ...) outperforms

* update t1.rb including numeric sort; fix appropriateness of -t=

* more use of restrict pointers ... ?

================================================================
PYTHON
* pgr + stats_m same I/O modules??

================================================================
FYI

Semantic versioning:
Given a version number MAJOR.MINOR.PATCH, increment the:

* MAJOR version when you make incompatible API changes,
* MINOR version when you add functionality in a backwards-compatible manner, and
* PATCH version when you make backwards-compatible bug fixes.

Initial release: https://news.ycombinator.com/item?id=10066742
v2.0.0  release: https://news.ycombinator.com/item?id=10132831

HN FEEDBACKS 2015-08-15 (https://news.ycombinator.com/item?id=10066742).
look-ats:
* cq?
* https://github.com/harelba/q
* https://github.com/google/crush-tools
* https://github.com/BurntSushi/xsv
* https://github.com/pydata/pandas/blob/master/pandas/io/tests/test_parsers.py
* https://drill.apache.org
* https://github.com/dbro/csvquote

https://help.github.com/articles/github-flavored-markdown/

shell: mlr put '$z=$x+$y'
lldb:  run put "\$z=\$x+\$y"

http://include-what-you-use.org/

----------------------------------------------------------------
https://fedoraproject.org/wiki/How_to_create_an_RPM_package
https://wiki.centos.org/HowTos/Packages/ContributeYourRPMs
https://lists.centos.org/pipermail/centos/2012-September/129227.html
https://fedoraproject.org/wiki/Join_the_package_collection_maintainers
https://fedoraproject.org/wiki/How_to_get_sponsored_into_the_packager_group
https://fedoraproject.org/wiki/Package_Review_Process
https://docs.fedoraproject.org/ro/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch11s03.html
http://wiki.networksecuritytoolkit.org/nstwiki/index.php/HowTo_Create_A_Patch_File_For_A_RPM

================================================================
git remote add upstream https://github.com/Homebrew/homebrew
git fetch upstream
git rebase upstream/master
shasum -a 256 ../mlr-3.2.2.tar.gz
git diff HEAD^  HEAD
git diff HEAD^2 HEAD

----------------------------------------------------------------
git remote add upstream https://github.com/Homebrew/homebrew
git fetch upstream
git rebase upstream/master
git checkout -b miller-3.4.0
shasum -a 256 ../mlr-3.4.0.tar.gz
edit Library/Formula/miller.rb
git add ...
git commit -m 'miller 3.4.0'
git push -u origin miller-3.4.0
submit the pull request

----------------------------------------------------------------
Squash commits by:
  brew update
  git checkout $YOUR_BRANCH
  git rebase --interactive origin/master
  mark each commit other than the first as "squash" or "fixup"
  git push -f

http://codeinthehole.com/writing/pull-requests-and-other-good-practices-for-teams-using-github/
