From 1102890bacdc10476647dfdacc4a4bab67c41cf2 Mon Sep 17 00:00:00 2001 From: John Kerl Date: Sun, 27 Sep 2020 21:33:38 -0400 Subject: [PATCH] data-examples.rst --- docs/data-examples.rst | 189 +++++++++++++++++++++++++++++++++++++- docs/data-examples.rst.in | 86 ++++++++++++++++- 2 files changed, 273 insertions(+), 2 deletions(-) diff --git a/docs/data-examples.rst b/docs/data-examples.rst index 3ed236335..e53ffb6f5 100644 --- a/docs/data-examples.rst +++ b/docs/data-examples.rst @@ -4,4 +4,191 @@ Data-diving examples ================================================================ -To be ported +flins data +---------------------------------------------------------------- + +The `flins.csv `_ file is some sample data obtained from https://support.spatialkey.com/spatialkey-sample-csv-data. + +Vertical-tabular format is good for a quick look at CSV data layout -- seeing what columns you have to work with: + +:: + + $ head -n 2 data/flins.csv | mlr --icsv --oxtab cat + county Seminole + tiv_2011 22890.55 + tiv_2012 20848.71 + line Residential + +A few simple queries: + +:: + + $ mlr --from data/flins.csv --icsv --opprint count-distinct -f county | head + county count + Seminole 1 + Miami Dade 2 + Palm Beach 1 + Highlands 2 + Duval 1 + St. Johns 1 + + $ mlr --from data/flins.csv --icsv --opprint count-distinct -f construction,line + +Categorization of total insured value: + +:: + + $ mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012 + tiv_2012_min tiv_2012_mean tiv_2012_max + 19757.910000 1061531.463750 2785551.630000 + + $ mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012 -g construction,line + + $ mlr --from data/flins.csv --icsv --oxtab stats1 -a p0,p10,p50,p90,p95,p99,p100 -f hu_site_deductible + hu_site_deductible_p0 + hu_site_deductible_p10 + hu_site_deductible_p50 + hu_site_deductible_p90 + hu_site_deductible_p95 + hu_site_deductible_p99 + hu_site_deductible_p100 + + $ mlr --from data/flins.csv --icsv --opprint stats1 -a p95,p99,p100 -f hu_site_deductible -g county then sort -f county | head + county hu_site_deductible_p95 hu_site_deductible_p99 hu_site_deductible_p100 + Duval - - - + Highlands - - - + Miami Dade - - - + Palm Beach - - - + Seminole - - - + St. Johns - - - + + $ mlr --from data/flins.csv --icsv --oxtab stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012 + tiv_2011_tiv_2012_corr 0.935363 + tiv_2011_tiv_2012_ols_m 1.089091 + tiv_2011_tiv_2012_ols_b 103095.523356 + tiv_2011_tiv_2012_ols_n 8 + tiv_2011_tiv_2012_r2 0.874904 + + $ mlr --from data/flins.csv --icsv --opprint stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012 -g county + county tiv_2011_tiv_2012_corr tiv_2011_tiv_2012_ols_m tiv_2011_tiv_2012_ols_b tiv_2011_tiv_2012_ols_n tiv_2011_tiv_2012_r2 + Seminole - - - 1 - + Miami Dade 1.000000 0.930643 -2311.154328 2 1.000000 + Palm Beach - - - 1 - + Highlands 1.000000 1.055693 -4529.793939 2 1.000000 + Duval - - - 1 - + St. Johns - - - 1 - + +Color/shape data +---------------------------------------------------------------- + +The `colored-shapes.dkvp `_ file is some sample data produced by the `mkdat2 `_ script. The idea is: + +* Produce some data with known distributions and correlations, and verify that Miller recovers those properties empirically. +* Each record is labeled with one of a few colors and one of a few shapes. +* The ``flag`` field is 0 or 1, with probability dependent on color +* The ``u`` field is plain uniform on the unit interval. +* The ``v`` field is the same, except tightly correlated with ``u`` for red circles. +* The ``w`` field is autocorrelated for each color/shape pair. +* The ``x`` field is boring Gaussian with mean 5 and standard deviation about 1.2, with no dependence on color or shape. + +Peek at the data: + +:: + + $ wc -l data/colored-shapes.dkvp + 10078 data/colored-shapes.dkvp + + $ head -n 6 data/colored-shapes.dkvp | mlr --opprint cat + color shape flag i u v w x + yellow triangle 1 11 0.6321695890307647 0.9887207810889004 0.4364983936735774 5.7981881667050565 + red square 1 15 0.21966833570651523 0.001257332190235938 0.7927778364718627 2.944117399716207 + red circle 1 16 0.20901671281497636 0.29005231936593445 0.13810280912907674 5.065034003400998 + red square 0 48 0.9562743938458542 0.7467203085342884 0.7755423050923582 7.117831369597269 + purple triangle 0 51 0.4355354501763202 0.8591292672156728 0.8122903963006748 5.753094629505863 + red square 0 64 0.2015510269821953 0.9531098083420033 0.7719912015786777 5.612050466474166 + +Look at uncategorized stats (using `creach `_ for spacing). + +Here it looks reasonable that ``u`` is unit-uniform; something's up with ``v`` but we can't yet see what: + +:: + + $ mlr --oxtab stats1 -a min,mean,max -f flag,u,v data/colored-shapes.dkvp | creach 3 + flag_min 0 + flag_mean 0.398889 + flag_max 1 + + u_min 0.000044 + u_mean 0.498326 + u_max 0.999969 + + v_min -0.092709 + v_mean 0.497787 + v_max 1.072500 + +The histogram shows the different distribution of 0/1 flags: + +:: + + $ mlr --opprint histogram -f flag,u,v --lo -0.1 --hi 1.1 --nbins 12 data/colored-shapes.dkvp + bin_lo bin_hi flag_count u_count v_count + -0.100000 0.000000 6058 0 36 + 0.000000 0.100000 0 1062 988 + 0.100000 0.200000 0 985 1003 + 0.200000 0.300000 0 1024 1014 + 0.300000 0.400000 0 1002 991 + 0.400000 0.500000 0 989 1041 + 0.500000 0.600000 0 1001 1016 + 0.600000 0.700000 0 972 962 + 0.700000 0.800000 0 1035 1070 + 0.800000 0.900000 0 995 993 + 0.900000 1.000000 4020 1013 939 + 1.000000 1.100000 0 0 25 + +Look at univariate stats by color and shape. In particular, color-dependent flag probabilities pop out, aligning with their original Bernoulli probablities from the data-generator script: + +:: + + $ mlr --opprint stats1 -a min,mean,max -f flag,u,v -g color then sort -f color data/colored-shapes.dkvp + color flag_min flag_mean flag_max u_min u_mean u_max v_min v_mean v_max + blue 0 0.584354 1 0.000044 0.517717 0.999969 0.001489 0.491056 0.999576 + green 0 0.209197 1 0.000488 0.504861 0.999936 0.000501 0.499085 0.999676 + orange 0 0.521452 1 0.001235 0.490532 0.998885 0.002449 0.487764 0.998475 + purple 0 0.090193 1 0.000266 0.494005 0.999647 0.000364 0.497051 0.999975 + red 0 0.303167 1 0.000671 0.492560 0.999882 -0.092709 0.496535 1.072500 + yellow 0 0.892427 1 0.001300 0.497129 0.999923 0.000711 0.510627 0.999919 + + $ mlr --opprint stats1 -a min,mean,max -f flag,u,v -g shape then sort -f shape data/colored-shapes.dkvp + shape flag_min flag_mean flag_max u_min u_mean u_max v_min v_mean v_max + circle 0 0.399846 1 0.000044 0.498555 0.999923 -0.092709 0.495524 1.072500 + square 0 0.396112 1 0.000188 0.499385 0.999969 0.000089 0.496538 0.999975 + triangle 0 0.401542 1 0.000881 0.496859 0.999661 0.000717 0.501050 0.999995 + +Look at bivariate stats by color and shape. In particular, ``u,v`` pairwise correlation for red circles pops out: + +:: + + $ mlr --opprint --right stats2 -a corr -f u,v,w,x data/colored-shapes.dkvp + u_v_corr w_x_corr + 0.133418 -0.011320 + + $ mlr --opprint --right stats2 -a corr -f u,v,w,x -g color,shape then sort -nr u_v_corr data/colored-shapes.dkvp + color shape u_v_corr w_x_corr + red circle 0.980798 -0.018565 + orange square 0.176858 -0.071044 + green circle 0.057644 0.011795 + red square 0.055745 -0.000680 + yellow triangle 0.044573 0.024605 + yellow square 0.043792 -0.044623 + purple circle 0.035874 0.134112 + blue square 0.032412 -0.053508 + blue triangle 0.015356 -0.000608 + orange circle 0.010519 -0.162795 + red triangle 0.008098 0.012486 + purple triangle 0.005155 -0.045058 + purple square -0.025680 0.057694 + green square -0.025776 -0.003265 + orange triangle -0.030457 -0.131870 + yellow circle -0.064773 0.073695 + blue circle -0.102348 -0.030529 + green triangle -0.109018 -0.048488 diff --git a/docs/data-examples.rst.in b/docs/data-examples.rst.in index 23e95b03b..cfc49795d 100644 --- a/docs/data-examples.rst.in +++ b/docs/data-examples.rst.in @@ -1,4 +1,88 @@ Data-diving examples ================================================================ -To be ported +flins data +---------------------------------------------------------------- + +The `flins.csv `_ file is some sample data obtained from https://support.spatialkey.com/spatialkey-sample-csv-data. + +Vertical-tabular format is good for a quick look at CSV data layout -- seeing what columns you have to work with: + +:: + +POKI_RUN_COMMAND{{head -n 2 data/flins.csv | mlr --icsv --oxtab cat}}HERE + +A few simple queries: + +:: + +POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --opprint count-distinct -f county | head}}HERE + +POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --opprint count-distinct -f construction,line}}HERE + +Categorization of total insured value: + +:: + +POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012}}HERE + +POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --opprint stats1 -a min,mean,max -f tiv_2012 -g construction,line}}HERE + +POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --oxtab stats1 -a p0,p10,p50,p90,p95,p99,p100 -f hu_site_deductible}}HERE + +POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --opprint stats1 -a p95,p99,p100 -f hu_site_deductible -g county then sort -f county | head}}HERE + +POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --oxtab stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012}}HERE + +POKI_RUN_COMMAND{{mlr --from data/flins.csv --icsv --opprint stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012 -g county}}HERE + +Color/shape data +---------------------------------------------------------------- + +The `colored-shapes.dkvp `_ file is some sample data produced by the `mkdat2 `_ script. The idea is: + +* Produce some data with known distributions and correlations, and verify that Miller recovers those properties empirically. +* Each record is labeled with one of a few colors and one of a few shapes. +* The ``flag`` field is 0 or 1, with probability dependent on color +* The ``u`` field is plain uniform on the unit interval. +* The ``v`` field is the same, except tightly correlated with ``u`` for red circles. +* The ``w`` field is autocorrelated for each color/shape pair. +* The ``x`` field is boring Gaussian with mean 5 and standard deviation about 1.2, with no dependence on color or shape. + +Peek at the data: + +:: + +POKI_RUN_COMMAND{{wc -l data/colored-shapes.dkvp}}HERE + +POKI_RUN_COMMAND{{head -n 6 data/colored-shapes.dkvp | mlr --opprint cat}}HERE + +Look at uncategorized stats (using `creach `_ for spacing). + +Here it looks reasonable that ``u`` is unit-uniform; something's up with ``v`` but we can't yet see what: + +:: + +POKI_RUN_COMMAND{{mlr --oxtab stats1 -a min,mean,max -f flag,u,v data/colored-shapes.dkvp | creach 3}}HERE + +The histogram shows the different distribution of 0/1 flags: + +:: + +POKI_RUN_COMMAND{{mlr --opprint histogram -f flag,u,v --lo -0.1 --hi 1.1 --nbins 12 data/colored-shapes.dkvp}}HERE + +Look at univariate stats by color and shape. In particular, color-dependent flag probabilities pop out, aligning with their original Bernoulli probablities from the data-generator script: + +:: + +POKI_RUN_COMMAND{{mlr --opprint stats1 -a min,mean,max -f flag,u,v -g color then sort -f color data/colored-shapes.dkvp}}HERE + +POKI_RUN_COMMAND{{mlr --opprint stats1 -a min,mean,max -f flag,u,v -g shape then sort -f shape data/colored-shapes.dkvp}}HERE + +Look at bivariate stats by color and shape. In particular, ``u,v`` pairwise correlation for red circles pops out: + +:: + +POKI_RUN_COMMAND{{mlr --opprint --right stats2 -a corr -f u,v,w,x data/colored-shapes.dkvp}}HERE + +POKI_RUN_COMMAND{{mlr --opprint --right stats2 -a corr -f u,v,w,x -g color,shape then sort -nr u_v_corr data/colored-shapes.dkvp}}HERE