Mirrors/miller
1
0
Fork 0
mirror of https://github.com/johnkerl/miller.git synced 2026-07-18 00:45:47 +00:00
Code Issues Projects Releases Wiki Activity

to-do's and style neatens

Browse source
This commit is contained in:
John Kerl 2015-05-07 19:22:23 -07:00
parent c1dcf4a507
commit d43a1eef0b
19 changed files with 391 additions and 79 deletions
Download patch file Download diff file Expand all files Collapse all files

13
c/mapping/mapper_sort.c
View file

@ -31,6 +31,19 @@ sllv_t* mapper_sort_func(lrec_t* pinrec, context_t* pctx, void* pvstate) {
plist = sllv_alloc();
sllv_add(plist, pinrec);
lhmslv_put(pstate->precords_by_key_field_names, slls_copy(pkey_field_values), plist);
// need num/lex & +/- flags for the sort -- static context for the non-reentrant qsort
// 'v' payload should be a pair of:
// * union of double & char*: here do the sscanf of the double if doing numeric sort on that field || abend
// * record-list as now
// at the end make an array of the pairs and qsort them on the union part
// then emit the record-lists as now
//
// before doing that, do the num/lex & the +/- at the CLI interface & handle (inefficiently)
// via repeated sscanf in the comparator:
// -rn a,b -fn c,d,e -r f,g -f h,i,j,k -- OR--
// -nr a,b -nf c,d,e -r f,g -f h,i,j,k
//
// data structure down there: sllv of sort info which is triple of char* field_name, char do_num, char do_rev.
} else {
sllv_add(plist, pinrec);
}

16
c/todo.txt
View file

@ -1,15 +1,20 @@
================================================================
! BUGFIX !
--ofmt ignored in put. perhaps best to reglobalize.
================================================================
FEATURES
!! sort -nr !!
!! quantiles !!
!! reminder pgr legend is broken !!
http://en.wikipedia.org/wiki/Order_statistic_tree
* stats1 mode
* stats1 mode: lhmsi & then sort. what about "1"=="1.0"?
! mod op!! (c-like, or sane) and put into wikidoc if so.
* sub function. e.g. "300ms" -> "300"
* online help: have a categorized function lister -- by string/math or arity, or some such ...
* nidx feature: bulk rename cols 1,2,3,4,5 this,that,something,foo,bar
* feature: ordered cut (a la reorder)
* impl mlr check: no output
* bulk rename cols 1,2,3,4,5 this,that,something,foo,bar
* ordered cut (a la reorder). either a new command (yeck) or cut option (e.g. cut -o)
* impl/doc mlr check: no output
* linreq-quality 2nd pass -- code it up in stats2 w/ -m {m} -b {b} -- ?
================================================================
@ -30,6 +35,7 @@ NEATEN
ONLINE HELP
* then-chaining note into mlr online help
* jko mlrdoc & gh/jk/mlr urls into mlr online help
* put/filter: have a categorized function lister -- by string/math or arity, or some such ...
================================================================
IMPROVEMENTS

35
c/tools/gen-squares.c
View file

@ -47,6 +47,8 @@ static void usage(char* argv0) {
int main(int argc, char** argv) {
double chomp = 0.5;
int n = 100000;
int mash_x = 0;
int mash_y = 0;
if (argc == 2) {
if (sscanf(argv[1], "%lf", &chomp) != 1)
usage(argv[0]);
@ -55,6 +57,15 @@ int main(int argc, char** argv) {
usage(argv[0]);
if (sscanf(argv[2], "%d", &n) != 1)
usage(argv[0]);
} else if (argc == 5) {
if (sscanf(argv[1], "%lf", &chomp) != 1)
usage(argv[0]);
if (sscanf(argv[2], "%d", &n) != 1)
usage(argv[0]);
if (sscanf(argv[3], "%d", &mash_x) != 1)
usage(argv[0]);
if (sscanf(argv[4], "%d", &mash_y) != 1)
usage(argv[0]);
} else {
usage(argv[0]);
}
@ -67,6 +78,18 @@ int main(int argc, char** argv) {
double x = (double)rand() / (double)RAND_MAX;
double y = (double)rand() / (double)RAND_MAX;
if ((x < lo && y < lo) || (x > hi && y > hi)) {
if (mash_x) {
if (x < lo)
x = 0.0;
else
x = 1.0;
}
if (mash_y) {
if (y < lo)
y = 0.0;
else
y = 1.0;
}
i++;
printf("x=%.18lf,y=%.18lf\n", x, y);
}
@ -78,6 +101,18 @@ int main(int argc, char** argv) {
double x = (double)rand() / (double)RAND_MAX;
double y = (double)rand() / (double)RAND_MAX;
if ((x < lo && y > hi) || (x > hi && y < lo)) {
if (mash_x) {
if (x < lo)
x = 0.0;
else
x = 1.0;
}
if (mash_y) {
if (y < lo)
y = 0.0;
else
y = 1.0;
}
i++;
printf("x=%.18lf,y=%.18lf\n", x, y);
}

16
doc/build.html
View file

@ -21,7 +21,21 @@ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}</script>
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis

16
doc/contact.html
View file

@ -21,7 +21,21 @@ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}</script>
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis

3
doc/content-for-file-formats.html
View file

@ -1,6 +1,7 @@
POKI_PUT_TOC_HERE
<h1>CSV/TSV</h1> When <tt>mlr</tt> is invoked with the <tt>--csv</tt> option,
<h1>CSV/TSV</h1>
When <tt>mlr</tt> is invoked with the <tt>--csv</tt> option,
key names are found on the first record and values are taken from subsequent
records. This includes the case of CSV-formatted files. See
POKI_PUT_LINK_FOR_PAGE(record-heterogeneity.html)HERE for how Miller handles

10
doc/css/miller.css
View file

@ -31,20 +31,24 @@ titleinbody {
color: maroon;
font-size: 120%
}
h1 {
font-weight: bold;
color: maroon;
font-size: 110%
font-size: 110%;
border-bottom: 1px solid #aaaaaa;
}
h2 {
font-weight: bold;
color: maroon;
font-size: 105%
font-size: 105%;
border-bottom: 1px solid #aaaaaa;
}
h3 {
font-weight: bold;
color: maroon;
font-size: 110%
font-size: 100%;
border-bottom: 1px solid #aaaaaa;
}
table {

25
doc/data-examples.html
View file

@ -21,7 +21,21 @@ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}</script>
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis
@ -76,7 +90,8 @@ Miller commands were run with pretty-print-tabular output format.
&bull;&nbsp;<a href="#Program_timing">Program timing</a><br/>
</div>
<p/>
<h1>flins data</h1> <a id="flins_data"/>
<a id="flins_data"/><h1>flins data</h1>
<p/> The <a href="data/flins.csv">flins.csv</a> file is some sample data
obtained from <a href="https://support.spatialkey.com/spatialkey-sample-csv-data">https://support.spatialkey.com/spatialkey-sample-csv-data</a>.
@ -176,7 +191,8 @@ cat flins.csv | mlr --icsv --opprint stats2 -a corr,linreg-ols,r2 -f tiv_2011,ti
<p/> xxx plaintext hardlinks
<h1>Color/shape data</h1> <a id="Color/shape_data"/>
<a id="Color/shape_data"/><h1>Color/shape data</h1>
<p/> The <a href="data/colored-shapes.dkvp">colored-shapes.dkvp</a> file is some sample data produced by the
<a href="https://github.com/johnkerl/miller/blob/master/doc/datagen/mkdat2">mkdat2</a> script. The idea is
<ul>
@ -322,7 +338,8 @@ orange circle 0.021871 -0.040626
</div>
<p/>
<h1>Program timing</h1> <a id="Program_timing"/>
<a id="Program_timing"/><h1>Program timing</h1>
This admittedly artificial example demonstrates using Miller time and stats
functions to introspectly acquire some information about Miller&rsquo;s own
runtime. The <tt>delta</tt> function computes the difference between successive

16
doc/etymology.html
View file

@ -21,7 +21,21 @@ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}</script>
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis

25
doc/feature-comparison.html
View file

@ -21,7 +21,21 @@ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}</script>
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis
@ -77,7 +91,8 @@ Miller commands were run with pretty-print-tabular output format.
</div>
<p/>
<h1>File-format awareness</h1> <a id="File-format_awareness"/>
<a id="File-format_awareness"/><h1>File-format awareness</h1>
Miller respects CSV headers. If you do <tt>mlr --csv-input cat *.csv</tt> then the header line is written once:
<table><tr>
@ -121,7 +136,8 @@ a,b,c
Likewise with <tt>mlr sort</tt>, <tt>mlr tac</tt>, and so on.
<h1>awk-like features: mlr filter and mlr put</h1> <a id="awk-like_features:_mlr_filter_and_mlr_put"/>
<a id="awk-like_features:_mlr_filter_and_mlr_put"/><h1>awk-like features: mlr filter and mlr put</h1>
<ul>
<li/> <tt>mlr filter</tt> includes/excludes records based on a filter
@ -151,7 +167,8 @@ startup time, with the bulk data-stream processing all done in C.
</ul>
<h1>See also</h1> <a id="See_also"/>
<a id="See_also"/><h1>See also</h1>
<p/>See <a href="reference.html">Reference</a> for more on Miller&rsquo;s
subcommands <tt>cat</tt>, <tt>cut</tt>, <tt>head</tt>, <tt>sort</tt>,
<tt>tac</tt>, <tt>tail</tt>, <tt>top</tt>, and <tt>uniq</tt>, as well as awk-like

32
doc/file-formats.html
View file

@ -21,7 +21,21 @@ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}</script>
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis
@ -79,7 +93,9 @@ Miller commands were run with pretty-print-tabular output format.
</div>
<p/>
<h1>CSV/TSV</h1> When <tt>mlr</tt> is invoked with the <tt>--csv</tt> option, <a id="CSV/TSV"/>key names are found on the first record and values are taken from subsequent
<a id="CSV/TSV"/><h1>CSV/TSV</h1>
When <tt>mlr</tt> is invoked with the <tt>--csv</tt> option,
key names are found on the first record and values are taken from subsequent
records. This includes the case of CSV-formatted files. See
<a href="record-heterogeneity.html">Record-heterogeneity</a> for how Miller handles
changes of field names within a single data stream.
@ -89,7 +105,8 @@ just as <tt>awk</tt> does. For TSV, use <tt>--fs TAB</tt>; to convert TSV to
CSV, use <tt>--ifs TAB --ofs ,</tt> etc. (See also
<a href="reference.html">Reference</a>.)
<h1>Pretty-printed</h1> <a id="Pretty-printed"/>Miller&rsquo;s pretty-print format is like CSV, but column-aligned. For example, compare
<a id="Pretty-printed"/><h1>Pretty-printed</h1>
Miller&rsquo;s pretty-print format is like CSV, but column-aligned. For example, compare
<table><tr><td>
<p/>
@ -133,7 +150,8 @@ files is constrained by available machine memory.
<p/> See <a href="record-heterogeneity.html">Record-heterogeneity</a> for how Miller
handles changes of field names within a single data stream.
<h1>Key-value pairs</h1> <a id="Key-value_pairs"/>Miller&rsquo;s default file format is DKVP, for <b>delimited key-value pairs</b>. Example:
<a id="Key-value_pairs"/><h1>Key-value pairs</h1>
Miller&rsquo;s default file format is DKVP, for <b>delimited key-value pairs</b>. Example:
<p/>
<div class="pokipanel">
@ -175,7 +193,8 @@ to analyze my logs.
<p/>See <a href="reference.html">Reference</a> regarding how to specify separators other than
the default equals-sign and comma.
<h1>Index-numbered (toolkit style)</h1> <a id="Index-numbered_(toolkit_style)"/>
<a id="Index-numbered_(toolkit_style)"/><h1>Index-numbered (toolkit style)</h1>
With <tt>--inidx --ifs ' ' --repifs</tt>, Miller splits lines on whitespace and
assigns integer field names starting with 1. This recapitulates Unix-toolkit
behavor.
@ -262,7 +281,8 @@ light
<p/>
</td> </tr></table>
<h1>Vertical tabular</h1> <a id="Vertical_tabular"/>
<a id="Vertical_tabular"/><h1>Vertical tabular</h1>
<p/>This is perhaps most useful for looking a very wide and/or multi-column
data which causes line-wraps on the screen (but see also <a
href="https://github.com/twosigma/ngrid">https://github.com/twosigma/ngrid</a>

16
doc/index.html
View file

@ -21,7 +21,21 @@ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}</script>
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis

16
doc/originality.html
View file

@ -21,7 +21,21 @@ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}</script>
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis

31
doc/performance.html
View file

@ -21,7 +21,21 @@ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}</script>
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis
@ -79,7 +93,8 @@ Miller commands were run with pretty-print-tabular output format.
</div>
<p/>
<h1>Data</h1> <a id="Data"/>
<a id="Data"/><h1>Data</h1>
Test data were of the form
<table><tr><td>
@ -119,7 +134,8 @@ DKVP) were used. In experiments not shown here, I also varied the file sizes;
the size-dependent results were the expected, completely unsurprising
linearities and so I produced no file-size-dependent plots for your viewing pleasure.
<h1>Comparands</h1> <a id="Comparands"/>
<a id="Comparands"/><h1>Comparands</h1>
The <tt>cat</tt>, <tt>cut</tt>, <tt>awk</tt>, <tt>sed</tt>, <tt>sort</tt> tools
were compared to <tt>mlr</tt> on an 8-core Darwin laptop; RAM capacity was
nowhere near challenged . The <tt>catc</tt> program is a simple line-oriented
@ -128,7 +144,8 @@ href="https://github.com/johnkerl/miller/blob/master/c/tools/catc.c">source
here</a>) which is intermediate between Miller (which is record-aware as well
as line-aware) and <tt>cat</tt> (which is only byte-aware).
<h1>Raw results</h1> <a id="Raw_results"/>
<a id="Raw_results"/><h1>Raw results</h1>
Note that for CSV data, the command is <tt>mlr --csv ...</tt> rather than <tt>mlr ...</tt>.
<p/>
@ -167,7 +184,8 @@ Note that for CSV data, the command is <tt>mlr --csv ...</tt> rather than <tt>ml
</div>
<p/>
<h1>Analysis</h1> <a id="Analysis"/>
<a id="Analysis"/><h1>Analysis</h1>
<ul>
<li/> As expected, <tt>cat</tt> is very fast &mdash; it needs only stream bytes as quickly as possible; it doesn&rsquo;t even need to touch individual bytes.
<li/> My <tt>catc</tt> is also faster than Miller: it needs to read and write lines, but it doesn&rsquo;t segment lines into records; in fact it does no iteration over bytes in each line.
@ -180,7 +198,8 @@ compare to other tools wherein such computations are less straightforward;
rather, I attempted only to show that Miller&rsquo;s processing time here is comparable to its own processing time for other problems.
</ul>
<h1>Conclusion</h1> <a id="Conclusion"/>
<a id="Conclusion"/><h1>Conclusion</h1>
For record-oriented data transformations, Miller is worth consideration. Field renames are worth doing as a pre-pipe or post-pipe using <tt>sed</tt>.
</td>

28
doc/record-heterogeneity.html
View file

@ -21,7 +21,21 @@ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}</script>
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis
@ -82,8 +96,10 @@ We think of CSV tables as rectangular: if there are 17 columns in the header the
<p/>But heterogeneous data abound (today&rsquo;s no-SQL databases for example). Miller handles this.
<h1>For I/O</h1> <a id="For_I/O"/>
<h2>CSV and pretty-print</h2> <a id="CSV_and_pretty-print"/>
<a id="For_I/O"/><h1>For I/O</h1>
<a id="CSV_and_pretty-print"/><h2>CSV and pretty-print</h2>
Miller simply prints a newline and a new header when there is a schema change. When there is no schema change, you get standard CSV as a special case. Likewise, Miller reads heterogeneous CSV or pretty-print input the same way. For example:
<table><tr><td>
@ -181,7 +197,8 @@ record_count resource
<p/>
</td></tr></table>
<h2>Key-value-pair, vertical-tabular, and index-numbered formats</h2> <a id="Key-value-pair,_vertical-tabular,_and_index-numbered_formats"/>
<a id="Key-value-pair,_vertical-tabular,_and_index-numbered_formats"/><h2>Key-value-pair, vertical-tabular, and index-numbered formats</h2>
For these formats, record-heterogeneity comes naturally:
<table><tr><td>
@ -262,7 +279,8 @@ resource /path/to/second/file
<p/>
</td></tr></table>
<h1>For processing</h1> <a id="For_processing"/>
<a id="For_processing"/><h1>For processing</h1>
<p/> Miller operates on specified fields and takes the rest along: for example, if you are sorting on the
<tt>count</tt> field then all records in the input stream must have a <tt>count</tt> field but the other fields can
vary, and moreover the sorted-on field name(s) don&rsquo;t need to be in the same position on each line:

103
doc/reference.html
View file

@ -21,7 +21,21 @@ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}</script>
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis
@ -103,7 +117,8 @@ Miller commands were run with pretty-print-tabular output format.
</div>
<p/>
<h1>Command overview</h1> <a id="Command_overview"/>
<a id="Command_overview"/><h1>Command overview</h1>
<p>
Whereas the Unix toolkit is made of the separate executables <tt>cat</tt>, <tt>tail</tt>, <tt>cut</tt>,
<tt>sort</tt>, etc., Miller has subcommands, invoked as follows:
@ -182,7 +197,8 @@ mlr sort hostname,uptime *.dat
</table>
<h1>On-line help</h1> <a id="On-line_help"/>
<a id="On-line_help"/><h1>On-line help</h1>
<p/>Examples:<p/>
<p/>
@ -218,7 +234,8 @@ Usage: mlr sort {comma-separated field names}
</div>
<p/>
<h1>then-chaining</h1> <a id="then-chaining"/>
<a id="then-chaining"/><h1>then-chaining</h1>
<p/>
In accord with the
<a href="http://en.wikipedia.org/wiki/Unix_philosophy">Unix philosophy</a>, you can pipe data into or out of
@ -246,9 +263,11 @@ mlr cut --complement -f os_version then sort hostname,uptime *.dat
<p/>
<!-- ================================================================ -->
<h1>I/O options</h1> <a id="I/O_options"/>
<a id="I/O_options"/><h1>I/O options</h1>
<!-- ================================================================ -->
<h2>Formats</h2> <a id="Formats"/>
<a id="Formats"/><h2>Formats</h2>
<p/> Options:
<pre>
@ -306,7 +325,8 @@ wye pan 5 0.5732889198020006 0.8636244699032729
</ul>
<!-- ================================================================ -->
<h2>Record/field/pair separators</h2> <a id="Record/field/pair_separators"/>
<a id="Record/field/pair_separators"/><h2>Record/field/pair separators</h2>
<p/> Miller has record separators <tt>IRS</tt> and <tt>ORS</tt>, field
separators <tt>IFS</tt> and <tt>OFS</tt>, and pair separators <tt>IPS</tt> and
<tt>OPS</tt>. For example, in the DKVP line <tt>a=1,b=2,c=3</tt>, the record
@ -345,7 +365,8 @@ may use the symbolic names <tt>newline</tt>, <tt>space</tt>, <tt>tab</tt>,
</ul>
<!-- ================================================================ -->
<h2>Number formatting</h2> <a id="Number_formatting"/>
<a id="Number_formatting"/><h2>Number formatting</h2>
Options:
<pre>
--ofmt {format string}
@ -370,9 +391,11 @@ embedded whitespace, which may not be what you want if you pipe the output to
something else.
<!-- ================================================================ -->
<h1>Data transformations</h1> <a id="Data_transformations"/>
<a id="Data_transformations"/><h1>Data transformations</h1>
<!-- ================================================================ -->
<h2>cat</h2> <a id="cat"/>
<a id="cat"/><h2>cat</h2>
<p/> Most useful for format conversions (see
<a href="file-formats.html">File formats</a>), and concatenating multiple
same-schema CSV files to have the same header:
@ -432,7 +455,8 @@ c 9
</td></tr></table>
<!-- ================================================================ -->
<h2>count-distinct</h2> <a id="count-distinct"/>
<a id="count-distinct"/><h2>count-distinct</h2>
<p/>
<div class="pokipanel">
<pre>
@ -479,7 +503,8 @@ a=zee,b=wye,count=455
<p/>
<!-- ================================================================ -->
<h2>cut</h2> <a id="cut"/>
<a id="cut"/><h2>cut</h2>
<p/>
<div class="pokipanel">
<pre>
@ -526,7 +551,8 @@ i x y
<p/>
<!-- ================================================================ -->
<h2>filter</h2> <a id="filter"/>
<a id="filter"/><h2>filter</h2>
<p/>
<div class="pokipanel">
<pre>
@ -610,7 +636,8 @@ x_y_corr
</td></tr></table>
<!-- ================================================================ -->
<h2>group-by</h2> <a id="group-by"/>
<a id="group-by"/><h2>group-by</h2>
<p/>
<div class="pokipanel">
<pre>
@ -670,7 +697,8 @@ they were encountered in the data stream, which in some cases may be more intere
to you.
<!-- ================================================================ -->
<h2>group-like</h2> <a id="group-like"/>
<a id="group-like"/><h2>group-like</h2>
<p/>
<div class="pokipanel">
<pre>
@ -717,7 +745,8 @@ record_count resource
</td></tr></table>
<!-- ================================================================ -->
<h2>having-fields</h2> <a id="having-fields"/>
<a id="having-fields"/><h2>having-fields</h2>
<p/>
<div class="pokipanel">
<pre>
@ -772,7 +801,8 @@ resource=/some/other/path,loadsec=0.97,ok=false
</td></tr></table>
<!-- ================================================================ -->
<h2>head</h2> <a id="head"/>
<a id="head"/><h2>head</h2>
<p/>
<div class="pokipanel">
<pre>
@ -818,7 +848,8 @@ wye hat 24 0.7286126830627567 0.19441962592638418
</td></tr></table>
<!-- ================================================================ -->
<h2>histogram</h2> <a id="histogram"/>
<a id="histogram"/><h2>histogram</h2>
<p/>
<div class="pokipanel">
<pre>
@ -859,7 +890,8 @@ bin_lo bin_hi x_count x2_count x3_count
<p/>
<!-- ================================================================ -->
<h2>put</h2> <a id="put"/>
<a id="put"/><h2>put</h2>
<p/>
<div class="pokipanel">
<pre>
@ -917,7 +949,8 @@ pan zee 10003 0.272054845593895200 0.425789896597056627 5 10 5 2 data/s
<p/>
<!-- ================================================================ -->
<h2>rename</h2> <a id="rename"/>
<a id="rename"/><h2>rename</h2>
<p/>
<div class="pokipanel">
<pre>
@ -992,7 +1025,8 @@ a=wye,b=pan,i=5,x=0.5732889198020006,COLUMN5=0.8636244699032729
</td></tr></table>
<!-- ================================================================ -->
<h2>reorder</h2> <a id="reorder"/>
<a id="reorder"/><h2>reorder</h2>
<p/>
<div class="pokipanel">
<pre>
@ -1055,7 +1089,8 @@ wye 0.5732889198020006 0.8636244699032729 5 pan
</td></tr></table>
<!-- ================================================================ -->
<h2>sort</h2> <a id="sort"/>
<a id="sort"/><h2>sort</h2>
<p/>
<div class="pokipanel">
<pre>
@ -1068,7 +1103,8 @@ Usage: mlr sort {comma-separated field names}
<p/>xxx write up after -n/-r.
<!-- ================================================================ -->
<h2>stats1</h2> <a id="stats1"/>
<a id="stats1"/><h2>stats1</h2>
<p/>
<div class="pokipanel">
<pre>
@ -1116,7 +1152,8 @@ zee 0.504242 0.502997
</td></tr></table>
<!-- ================================================================ -->
<h2>stats2</h2> <a id="stats2"/>
<a id="stats2"/><h2>stats2</h2>
<p/>
<div class="pokipanel">
<pre>
@ -1202,7 +1239,8 @@ plotting; here&rsquo;s a screenshot.
<p/> (Thanks Drew Kunas for a good conversation about PCA!)
<!-- ================================================================ -->
<h2>step</h2> <a id="step"/>
<a id="step"/><h2>step</h2>
<p/>
<div class="pokipanel">
<pre>
@ -1270,7 +1308,8 @@ eks zee 14 0.5207382318405251 0.34141681118811673 0.029221 2.764119
</td></tr></table>
<!-- ================================================================ -->
<h2>tac</h2> <a id="tac"/>
<a id="tac"/><h2>tac</h2>
<p/>
<div class="pokipanel">
<pre>
@ -1333,7 +1372,8 @@ a b c filename
</td></tr></table>
<!-- ================================================================ -->
<h2>tail</h2> <a id="tail"/>
<a id="tail"/><h2>tail</h2>
<p/>
<div class="pokipanel">
<pre>
@ -1375,7 +1415,8 @@ yellow triangle 1 99999 0.5369350176939407 0.5197619334387739 0.506446844647
</td></tr></table>
<!-- ================================================================ -->
<h2>top</h2> <a id="top"/>
<a id="top"/><h2>top</h2>
<p/>
<div class="pokipanel">
<pre>
@ -1430,7 +1471,8 @@ zee 2 0.999438
</td></tr></table>
<!-- ================================================================ -->
<h2>uniq</h2> <a id="uniq"/>
<a id="uniq"/><h2>uniq</h2>
<p/>
<div class="pokipanel">
<pre>
@ -1507,7 +1549,8 @@ yellow triangle 4667
</td></tr></table>
<!-- ================================================================ -->
<h1>Functions for filter and put</h1> <a id="Functions_for_filter_and_put"/>
<a id="Functions_for_filter_and_put"/><h1>Functions for filter and put</h1>
Miller&rsquo;s
<a href="#filter"><tt>filter</tt></a> and <a href="#put"><tt>put</tt></a>
support the following operators and functions:

16
doc/template.html
View file

@ -19,7 +19,21 @@ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}</script>
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis

16
doc/to-do.html
View file

@ -21,7 +21,21 @@ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}</script>
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis

37
doc/whyc.html
View file

@ -21,7 +21,21 @@ document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.
try {
var pageTracker = _gat._getTracker("UA-15651652-1");
pageTracker._trackPageview();
} catch(err) {}</script>
} catch(err) {}
</script>
<script type="text/javascript">
function toggle(divName) {
var eleDiv = document.getElementById(divName);
if (eleDiv != null) {
if (eleDiv.style.display == "block") {
eleDiv.style.display = "none";
} else {
eleDiv.style.display = "block";
}
}
}
</script>
<!--
The background image is from a screenshot of a Google search for "data analysis
@ -81,7 +95,8 @@ Miller commands were run with pretty-print-tabular output format.
</div>
<p/>
<h1>Why not C?</h1> <a id="Why_not_C?"/>
<a id="Why_not_C?"/><h1>Why not C?</h1>
<p/>C lacks many of the features found in modern, high-level languages such as
Java or Go: garbage collection, collections libraries, generics/near-generics,
hash-map/linked-list literals built into the language (e.g.
@ -99,7 +114,8 @@ Moreover, Miller&rsquo;s primary data structure, the
is hand-tuned to Miller&rsquo;s use case and would have required hand-coding in
any case.
<h1>C vs. Go, D, Rust, etc.; C is fast</h1> <a id="C_vs._Go,_D,_Rust,_etc.;_C_is_fast"/>
<a id="C_vs._Go,_D,_Rust,_etc.;_C_is_fast"/><h1>C vs. Go, D, Rust, etc.; C is fast</h1>
<p/>I love Go (<a href="http://golang.org">https://golang.org</a>): I think
it&rsquo;s one of the best things ever to happen to our craft, and I use it
often. The D language (<a href="http://dlang.org">http://dolang.org</a>) is an
@ -117,12 +133,14 @@ application domain &mdash; parallelism here is more easily achieved by running
multiple single-threaded processes, each handling its own input files, either
on a single host or split across multiple hosts.
<h1>C is ubiquitous</h1> <a id="C_is_ubiquitous"/>
<a id="C_is_ubiquitous"/><h1>C is ubiquitous</h1>
<p/>Every Unix-like system has a C compiler (or is an <tt>apt-get</tt> or
<tt>yum install</tt> away from it). This, I hope, bodes well for uptake
of Miller.
<h1>C is old-school</h1> <a id="C_is_old-school"/>
<a id="C_is_old-school"/><h1>C is old-school</h1>
<p/>This alone is not enough reason to program in C, but since I find myself
coding in C due to the other reasons on this page, it&rsquo;s happy enough to
use a throwback language for a throwback tool (see
@ -132,7 +150,8 @@ use of modern tools such as <a href="http://valgrind.org">valgrind</a>.
K&amp;R was a long, long time ago. (I&rsquo;m writing plain C with <tt>//</tt>
comments; enough said.)
<h1>C vs. C++</h1> <a id="C_vs._C++"/>
<a id="C_vs._C++"/><h1>C vs. C++</h1>
I have a strong personal distaste for C++: its syntax is an ugly layer over the
simplicity of C++; templates and STL are even more awkward and even less
elegant. (Meanwhile I find Java, Go, and D to be both elegant and modern; they
@ -140,7 +159,8 @@ were ruled out not for aesthetics but for performance as described above.)
Meanwhile all the positive features I would want from C++ are easily
implementable in C:
<h2><tt>this</tt> pointers and attributes</h2> <a id="<tt>this</tt>_pointers_and_attributes"/>The C++ compiler implictly inserts <tt>this</tt> pointers into method calls:
<a id="<tt>this</tt>_pointers_and_attributes"/><h2><tt>this</tt> pointers and attributes</h2>
The C++ compiler implictly inserts <tt>this</tt> pointers into method calls:
for example
<pre>
@ -208,7 +228,8 @@ int lrec_foo(lrec_t* prec) {
</pre>
<h2>Interfaces and virtual-function pointers</h2> <a id="Interfaces_and_virtual-function_pointers"/>
<a id="Interfaces_and_virtual-function_pointers"/><h2>Interfaces and virtual-function pointers</h2>
Coding conventions again do most of the work, here accompanied by typdeffed function pointers.
For example, here is Miller&rsquo;s record reader interface:
<pre>