From df4116bf33376038d5ddade0cdae54c9fe854eaa Mon Sep 17 00:00:00 2001 From: John Kerl Date: Tue, 26 May 2015 17:02:03 -0400 Subject: [PATCH] neaten --- c/todo.txt | 1 - doc/content-for-feature-comparison.html | 6 ++- doc/content-for-file-formats.html | 34 ++++++++++++++ doc/feature-comparison.html | 6 ++- doc/file-formats.html | 59 +++++++++++++++++++++++++ 5 files changed, 101 insertions(+), 5 deletions(-) diff --git a/c/todo.txt b/c/todo.txt index e84863141..a69763f84 100644 --- a/c/todo.txt +++ b/c/todo.txt @@ -7,7 +7,6 @@ ================================================================ FEATURES/+ -!! move lrec_parsers -> readers severally !! mmap impl: csv !! then: --mmap by default !! mmap/stdio UTs; maybe run all cases with --mmap and again with --no-mmap diff --git a/doc/content-for-feature-comparison.html b/doc/content-for-feature-comparison.html index 5bdfb908e..21faa9300 100644 --- a/doc/content-for-feature-comparison.html +++ b/doc/content-for-feature-comparison.html @@ -46,8 +46,10 @@ Its domain-specific languages are limited to the filter and functions are stream functions. This means NF, NR, etc. change from one line to another, $x is a label for field x in the current record, and the input to sqrt($x) changes from one record -to the next. Miller doesn’t let you set, say, sum=0 and then -update that on each record. +to the next. Miller doesn’t let you set sum=0 before +records are read, then update that sum on each record, then print its value at the +end. (However, do see mlr step -a rsum in the +POKI_PUT_LINK_FOR_PAGE(reference.html)HERE) page.)
  • Miller is faster than awk, cut, and so on (depending on platform; see also POKI_PUT_LINK_FOR_PAGE(performance.html)HERE). In diff --git a/doc/content-for-file-formats.html b/doc/content-for-file-formats.html index 0aa8bb75b..7b7d0c5c1 100644 --- a/doc/content-for-file-formats.html +++ b/doc/content-for-file-formats.html @@ -39,6 +39,40 @@ Miller’s default file format is DKVP, for delimited key-value pairs POKI_RUN_COMMAND{{mlr cat data/small}}HERE Such data are easy to generate, e.g. in Ruby with +POKI_CARDIFY(puts "host=#{hostname},seconds=#{t2-t1},message=#{msg}")HERE +POKI_CARDIFY(puts mymap.collect{|k,v| "#{k}=#{v}"}.join(','))HERE +or print statements in various languages, e.g. +POKI_CARDIFY(echo "type=3,user=$USER,date=$date\n";)HERE +POKI_CARDIFY(logger.log("type=3,user=$USER,date=$date\n");)HERE + +

    As discussed in POKI_PUT_LINK_FOR_PAGE(record-heterogeneity.html)HERE, Miller handles +changes of field names within the same data stream. But using DKVP format this is particularly +natural. One of my favorite use-cases for Miller is in application/server logs, where I log all sorts +of lines such as + +

    +

    +
    +resource=/path/to/file,loadsec=0.45,ok=true
+record_count=100, resource=/path/to/file
+resource=/some/other/path,loadsec=0.97,ok=false
+
    +
    +

    + +etc. and I just log them as needed. Then later, I can use grep, mlr --opprint group-like, etc. +to analyze my logs. + +

    See POKI_PUT_LINK_FOR_PAGE(reference.html)HERE regarding how to specify separators other than +the default equals-sign and comma. + +

    Index-numbered (toolkit style)

    + +With --inidx --ifs ' ' --repifs, Miller splits lines on whitespace and +assigns integer field names starting with 1. This recapitulates Unix-toolkit +behavor. + +

    Example with index-numbered output: 

       puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',')
    diff --git a/doc/feature-comparison.html b/doc/feature-comparison.html index e4e49a747..d12425e4e 100644 --- a/doc/feature-comparison.html +++ b/doc/feature-comparison.html @@ -170,8 +170,10 @@ Its domain-specific languages are limited to the filter and functions are stream functions. This means NF, NR, etc. change from one line to another, $x is a label for field x in the current record, and the input to sqrt($x) changes from one record -to the next. Miller doesn’t let you set, say, sum=0 and then -update that on each record. +to the next. Miller doesn’t let you set sum=0 before +records are read, then update that sum on each record, then print its value at the +end. (However, do see mlr step -a rsum in the +Reference) page.)
  • Miller is faster than awk, cut, and so on (depending on platform; see also Performance). In diff --git a/doc/file-formats.html b/doc/file-formats.html index cdb5e303b..84eaf8073 100644 --- a/doc/file-formats.html +++ b/doc/file-formats.html @@ -95,6 +95,7 @@ Miller commands were run with pretty-print-tabular output format. • Pretty-printed
    • Key-value pairs
    • Index-numbered (toolkit style)
    +• Index-numbered (toolkit style)
    • Vertical tabular

    @@ -173,6 +174,64 @@ a=wye,b=pan,i=5,x=0.5732889198020006,y=0.8636244699032729

    Such data are easy to generate, e.g. in Ruby with +

    +

    +
    +puts "host=#{hostname},seconds=#{t2-t1},message=#{msg}"
+
    +
    +

    +

    +

    +
    +puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',')
+
    +
    +

    +or print statements in various languages, e.g. +

    +

    +
    +echo "type=3,user=$USER,date=$date\n";
+
    +
    +

    +

    +

    +
    +logger.log("type=3,user=$USER,date=$date\n");
+
    +
    +

    + +

    As discussed in Record-heterogeneity, Miller handles +changes of field names within the same data stream. But using DKVP format this is particularly +natural. One of my favorite use-cases for Miller is in application/server logs, where I log all sorts +of lines such as + +

    +

    +
    +resource=/path/to/file,loadsec=0.45,ok=true
+record_count=100, resource=/path/to/file
+resource=/some/other/path,loadsec=0.97,ok=false
+
    +
    +

    + +etc. and I just log them as needed. Then later, I can use grep, mlr --opprint group-like, etc. +to analyze my logs. + +

    See Reference regarding how to specify separators other than +the default equals-sign and comma. + +

    Index-numbered (toolkit style)

    + +With --inidx --ifs ' ' --repifs, Miller splits lines on whitespace and +assigns integer field names starting with 1. This recapitulates Unix-toolkit +behavor. + +

    Example with index-numbered output: 

       puts mymap.collect{|k,v| "#{k}=#{v}"}.join(',')