Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-19 01:15:21 +00:00
Code Issues Projects Releases Wiki Activity
miller/doc/data-examples.html
John Kerl 1b49faa101 doc neaten
2015-10-05 20:23:02 -04:00

221 lines
No EOL
8 KiB
HTML
Raw Blame History

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html lang="en">
<!-- PAGE GENERATED FROM template.html and content-for-data-examples.html BY poki. -->
<!-- PLEASE MAKE CHANGES THERE AND THEN RE-RUN poki. -->
<head>
<meta http-equiv="Content-type" content="text/html;charset=UTF-8"/>
<meta name="description" content="Miller documentation"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0"/> <!-- mobile-friendly -->
<meta name="keywords"
content="John Kerl, Kerl, Miller, miller, mlr, OLAP, data analysis software, regression, correlation, variance, data tools, " />
<title> Data examples </title>
<link rel="stylesheet" type="text/css" href="css/miller.css"/>
<link rel="stylesheet" type="text/css" href="css/poki-callbacks.css"/>
</head>
<!-- ================================================================ -->
<script type="text/javascript">
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
</script>
<script type="text/javascript">
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis
tools", lightened and sepia-toned. Over this was placed a Mac Terminal app with
very light-grey font and translucent background, in which a few statistical
Miller commands were run with pretty-print-tabular output format.
-->
<body background="pix/sepia-overlay.jpg">
<!-- ================================================================ -->
<table width="100%">
<tr>
<!-- navbar -->
<td width="15%">
<!--
<img src="pix/mlr.jpg" />
<img style="border-width:1px; color:black;" src="pix/mlr.jpg" />
-->
<div class="pokinav">
<center><titleinbody>Miller</titleinbody></center>
<!-- PAGE LIST GENERATED FROM template.html BY poki -->
<br/>&bull;&nbsp;<a href="index.html">About Miller</a>
<br/>&bull;&nbsp;<a href="file-formats.html">File formats</a>
<br/>&bull;&nbsp;<a href="feature-comparison.html">Miller features in the context of the Unix toolkit</a>
<br/>&bull;&nbsp;<a href="record-heterogeneity.html">Record-heterogeneity</a>
<br/>&bull;&nbsp;<a href="reference.html">Reference</a>
<br/>&bull;&nbsp;<a href="data-examples.html"><b>Data examples</b></a>
<br/>&bull;&nbsp;<a href="internationalization.html">Internationalization</a>
<br/>&bull;&nbsp;<a href="build.html">Compiling, portability, dependencies, and testing</a>
<br/>&bull;&nbsp;<a href="performance.html">Performance</a>
<br/>&bull;&nbsp;<a href="whyc.html">Why C?</a>
<br/>&bull;&nbsp;<a href="etymology.html">Why call it Miller?</a>
<br/>&bull;&nbsp;<a href="originality.html">How original is Miller?</a>
<br/>&bull;&nbsp;<a href="to-do.html">Things to do</a>
<br/>&bull;&nbsp;<a href="contact.html">Contact information</a>
<br/>&bull;&nbsp;<a href="https://github.com/johnkerl/miller">GitHub repo</a>
<br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/>
<br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/>
<br/> <br/> <br/> <br/> <br/> <br/>
</div>
</td>
<!-- page body -->
<td>
<div style="overflow-y:scroll;height:1500px">
<center> <titleinbody> Data examples </titleinbody> </center>
<p/>
<!-- BODY COPIED FROM content-for-data-examples.html BY poki -->
<div class="pokitoc">
<center><b>Contents:</b></center>
&bull;&nbsp;<a href="#flins_data">flins data</a><br/>
&bull;&nbsp;<a href="#Color/shape_data">Color/shape data</a><br/>
&bull;&nbsp;<a href="#Program_timing">Program timing</a><br/>
</div>
<p/>
<a id="flins_data"/><h1>flins data</h1>
<p/> The <a href="data/flins.csv">flins.csv</a> file is some sample data
obtained from <a href="https://support.spatialkey.com/spatialkey-sample-csv-data">https://support.spatialkey.com/spatialkey-sample-csv-data</a>.
<p/>Vertical-tabular format is good for a quick look at CSV data layout &mdash; seeing what columns you have to work with:
<p/>
<div class="pokipanel">
<pre>
$ head -n 2 data/flins.csv | mlr --icsv --oxtab cat
policyID 119736
statecode FL
county CLAY COUNTY
eq_site_limit 498960
hu_site_limit 498960
fl_site_limit 498960
fr_site_limit 498960
tiv_2011 498960
tiv_2012 792148.9
eq_site_deductible 0
hu_site_deductible 9979.2
fl_site_deductible 0
fr_site_deductible 0
point_latitude 30.102261
point_longitude -81.711777
line Residential
construction Masonry
point_granularity 1
</pre>
</div>
<p/>
<p/> A few simple queries:
<p/>
<div class="pokipanel">
<pre>
$ cat data/flins.csv | mlr --icsv --opprint count-distinct -f county | head
county count
CLAY COUNTY 363
SUWANNEE COUNTY 154
NASSAU COUNTY 135
COLUMBIA COUNTY 125
ST JOHNS COUNTY 657
BAKER COUNTY 70
BRADFORD COUNTY 31
HAMILTON COUNTY 35
UNION COUNTY 15
</pre>
</div>
<p/>
<p/>
<div class="pokipanel">
<pre>
$ cat data/flins.csv | mlr --icsv --opprint count-distinct -f construction,line
construction line count
Masonry Residential 9257
Wood Residential 21581
Reinforced Concrete Commercial 1299
Reinforced Masonry Commercial 4225
Steel Frame Commercial 272
</pre>
</div>
<p/>
<p/> Categorization of total insured value:
<p/>
<div class="pokipanel">
<pre>
$ cat data/flins.csv | mlr --icsv --opprint stats1 -a min,mean,max -f tiv_2012
tiv_2012_min tiv_2012_mean tiv_2012_max
73.370000 2571004.097342 1701000000.000000
</pre>
</div>
<p/>
<p/>
<div class="pokipanel">
<pre>
$ cat data/flins.csv | mlr --icsv --opprint stats1 -a min,mean,max -f tiv_2012 -g construction,line
construction line tiv_2012_min tiv_2012_mean tiv_2012_max
Masonry Residential 261168.070000 1041986.129217 3234970.920000
Wood Residential 73.370000 113493.017049 649046.120000
Reinforced Concrete Commercial 6416016.010000 20212428.681840 60570000.000000
Reinforced Masonry Commercial 1287817.340000 4621372.981117 16650000.000000
Steel Frame Commercial 29790000.000000 133492500.000000 1701000000.000000
</pre>
</div>
<p/>
<p/>
<div class="pokipanel">
<pre>
$ cat data/flins.csv | mlr --icsv --oxtab stats1 -a p0,p10,p50,p90,p95,p99,p100 -f hu_site_deductible
hu_site_deductible_p0 0.000000
hu_site_deductible_p10 0.000000
hu_site_deductible_p50 0.000000
hu_site_deductible_p90 76.500000
hu_site_deductible_p95 6829.200000
hu_site_deductible_p99 126270.000000
hu_site_deductible_p100 7380000.000000
</pre>
</div>
<p/>
<p/>
<div class="pokipanel">
<pre>
$ cat data/flins.csv | mlr --icsv --opprint stats1 -a p95,p99,p100 -f hu_site_deductible -g county then sort -f county | head
county hu_site_deductible_p95 hu_site_deductible_p99 hu_site_deductible_p100
ALACHUA COUNTY 30630.600000 107312.400000 1641375.000000
BAKER COUNTY 0.000000 0.000000 0.000000
BAY COUNTY 26131.500000 181912.500000 630000.000000
BRADFORD COUNTY 3355.200000 8163.000000 8163.000000
BREVARD COUNTY 5360.400000 78975.000000 1973461.500000
BROWARD COUNTY 0.000000 148500.000000 3258900.000000
CALHOUN COUNTY 0.000000 33339.600000 33339.600000
CHARLOTTE COUNTY 5400.000000 52650.000000 250994.700000
CITRUS COUNTY 1332.900000 79974.900000 483785.100000
</pre>
</div>
<p/>
<p/>
<div class="pokipanel">
<pre>
$ cat data/flins.csv | mlr --icsv --oxtab stats2 -a corr,linreg-ols,r2 -f tiv_2011,tiv_2012
Reference in a new issue View git blame Copy permalink