neaten

.gitignore

@@ -83,6 +83,8 @@ data/.gitignore
 doc/jkopush
 doc/push2
 
+docs/_build
+
 c/mlr.static
 miller-*.src.rpm
 mlr.exe

docs/10min.rst

@@ -1,3 +1,6 @@
+..
+    PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
+
 Miller in 10 minutes
 ====================
 
@@ -392,6 +395,8 @@ We've also already seen other ways to write the same data::
 
 Anything we can do with CSV input data, we can do with any other format input data.  And you can read from one format, do any record-processing, and output to the same format as the input, or to a different output format.
 
+.. _sql-output-examples:
+
 SQL-output examples
 ^^^^^^^^^^^^^^^^^^^
 
@@ -480,6 +485,8 @@ Since Miller handles TSV output, we can do as much or as little processing as we
 
 Again, all the examples in the CSV section apply here -- just change the input-format flags.
 
+.. _sql-input-examples:
+
 SQL-input examples
 ^^^^^^^^^^^^^^^^^^
 
@@ -791,4 +798,4 @@ Alternatively, we can simply group the similar data for a better look::
 More
 ^^^^
 
-Please see the <a href="reference.html">reference</a> for complete information, as well as the <a href="faq.html">FAQ</a> and the <a href="cookbook.html">cookbook</a> for more tips.
+Please see the :doc:`reference` for complete information, as well as the :doc:`faq` and the :doc:`cookbook` for more tips.

docs/10min.rst.in

@@ -198,6 +198,8 @@ We've also already seen other ways to write the same data::
 
 Anything we can do with CSV input data, we can do with any other format input data.  And you can read from one format, do any record-processing, and output to the same format as the input, or to a different output format.
 
+.. _sql-output-examples:
+
 SQL-output examples
 ^^^^^^^^^^^^^^^^^^^
 
@@ -286,6 +288,8 @@ Since Miller handles TSV output, we can do as much or as little processing as we
 
 Again, all the examples in the CSV section apply here -- just change the input-format flags.
 
+.. _sql-input-examples:
+
 SQL-input examples
 ^^^^^^^^^^^^^^^^^^
 
@@ -442,4 +446,4 @@ POKI_RUN_COMMAND{{mlr --opprint group-like then sec2gmt time log.txt}}HERE
 More
 ^^^^
 
-Please see the <a href="reference.html">reference</a> for complete information, as well as the <a href="faq.html">FAQ</a> and the <a href="cookbook.html">cookbook</a> for more tips.
+Please see the :doc:`reference` for complete information, as well as the :doc:`faq` and the :doc:`cookbook` for more tips.

docs/_static/basic.css

@@ -42,6 +42,11 @@ div.related li {
     display: inline;
 }
 
+/* CHANGE ME */
+div.body li {
+    margin:0 0 10px 0;   
+}
+
 div.related li.right {
     float: right;
     margin-right: 5px;

docs/_static/classic.css

@@ -68,7 +68,8 @@ div.related {
 }
 
 div.related a {
-    color: #800000;
+    /* CHANGE ME */
+    color: #000000;
 }
 
 div.sphinxsidebar {
@@ -112,7 +113,7 @@ div.sphinxsidebar ul {
 
 div.sphinxsidebar a {
     /* CHANGE ME */
-    color: #808080;
+    color: #404040;
 }
 
 div.sphinxsidebar input {

docs/about.rst

@@ -1,3 +1,6 @@
+..
+    PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
+
 About
 =====
 
@@ -22,7 +25,7 @@ Features
 
 * Miller complements **data-analysis tools** such as **R**, **pandas**, etc.: you can use Miller to **clean** and **prepare** your data. While you can do **basic statistics** entirely in Miller, its streaming-data feature and single-pass algorithms enable you to **reduce very large data sets**.
 
-* Miller complements SQL **databases**: you can slice, dice, and reformat data on the client side on its way into or out of a database.  (Examples <a href="10-min.html#SQL-input_examples">here</a> and <a href="10-min.html#SQL-output_examples">here</a>). You can also reap some of the benefits of databases for quick, setup-free one-off tasks when you just need to query some data in disk files in a hurry.
+* Miller complements SQL **databases**: you can slice, dice, and reformat data on the client side on its way into or out of a database.  (Examples :ref:`here <sql-input-examples>` and :ref:`here <sql-output-examples>`.) You can also reap some of the benefits of databases for quick, setup-free one-off tasks when you just need to query some data in disk files in a hurry.
 
 * Miller also goes beyond the classic Unix tools by stepping fully into our modern, **no-SQL** world: its essential record-heterogeneity property allows Miller to operate on data where records with different schema (field names) are interleaved.
 
@@ -38,9 +41,9 @@ Features
 
 * Miller has high-throughput **performance** on par with the Unix toolkit
 
-* Not unlike <a href="http://stedolan.github.io/jq/">jq</a> (for JSON), Miller is written in portable, modern C, with **zero runtime dependencies**.  You can download or compile a single binary, ``scp`` it to a faraway machine, and expect it to work.
+* Not unlike `jq <https://stedolan.github.io/jq/>`_ (for JSON), Miller is written in portable, modern C, with **zero runtime dependencies**.  You can download or compile a single binary, ``scp`` it to a faraway machine, and expect it to work.
 
-Releases and release notes: <a href="https://github.com/johnkerl/miller/releases">https://github.com/johnkerl/miller/releases</a>.
+Releases and release notes: https://github.com/johnkerl/miller/releases.
 
 Examples
 ^^^^^^^^

docs/about.rst.in

@@ -0,0 +1,113 @@
+About
+=====
+
+Miller is like awk, sed, cut, join, and sort for **name-indexed data such as
+CSV, TSV, and tabular JSON**. You get to work with your data using named
+fields, without needing to count positional column indices.
+
+This is something the Unix toolkit always could have done, and arguably
+always should have done.  It operates on key-value-pair data while the familiar
+Unix tools operate on integer-indexed fields: if the natural data structure for
+the latter is the array, then Miller's natural data structure is the
+insertion-ordered hash map.  This encompasses a **variety of data formats**,
+including but not limited to the familiar CSV, TSV, and JSON.  (Miller can handle
+**positionally-indexed data** as a special case.)
+
+Features
+^^^^^^^^
+
+* Miller is **multi-purpose**: it's useful for **data cleaning**, **data reduction**, **statistical reporting**, **devops**, **system administration**, **log-file processing**, **format conversion**, and **database-query post-processing**.
+
+* You can use Miller to snarf and munge **log-file data**, including selecting out relevant substreams, then produce CSV format and load that into all-in-memory/data-frame utilities for further statistical and/or graphical processing.
+
+* Miller complements **data-analysis tools** such as **R**, **pandas**, etc.: you can use Miller to **clean** and **prepare** your data. While you can do **basic statistics** entirely in Miller, its streaming-data feature and single-pass algorithms enable you to **reduce very large data sets**.
+
+* Miller complements SQL **databases**: you can slice, dice, and reformat data on the client side on its way into or out of a database.  (Examples :ref:`here <sql-input-examples>` and :ref:`here <sql-output-examples>`.) You can also reap some of the benefits of databases for quick, setup-free one-off tasks when you just need to query some data in disk files in a hurry.
+
+* Miller also goes beyond the classic Unix tools by stepping fully into our modern, **no-SQL** world: its essential record-heterogeneity property allows Miller to operate on data where records with different schema (field names) are interleaved.
+
+* Miller is **streaming**: most operations need only a single record in memory at a time, rather than ingesting all input before producing any output.  For those operations which require deeper retention (``sort``, ``tac``, ``stats1``), Miller retains only as much data as needed.  This means that whenever functionally possible, you can operate on files which are larger than your system's available RAM, and you can use Miller in **tail -f** contexts.
+
+* Miller is **pipe-friendly** and interoperates with the Unix toolkit
+
+* Miller's I/O formats include **tabular pretty-printing**, **positionally indexed** (Unix-toolkit style), CSV, JSON, and others
+
+* Miller does **conversion** between formats
+
+* Miller's **processing is format-aware**: e.g. CSV ``sort`` and ``tac`` keep header lines first
+
+* Miller has high-throughput **performance** on par with the Unix toolkit
+
+* Not unlike `jq <https://stedolan.github.io/jq/>`_ (for JSON), Miller is written in portable, modern C, with **zero runtime dependencies**.  You can download or compile a single binary, ``scp`` it to a faraway machine, and expect it to work.
+
+Releases and release notes: https://github.com/johnkerl/miller/releases.
+
+Examples
+^^^^^^^^
+
+Column select::
+
+    % mlr --csv cut -f hostname,uptime mydata.csv
+
+Add new columns as function of other columns::
+
+    % mlr --nidx put '$sum = $7 < 0.0 ? 3.5 : $7 + 2.1*$8' *.dat
+
+Row filter::
+
+    % mlr --csv filter '$status != "down" && $upsec >= 10000' *.csv
+
+Apply column labels and pretty-print::
+
+    % grep -v '^#' /etc/group | mlr --ifs : --nidx --opprint label group,pass,gid,member then sort -f group
+
+Join multiple data sources on key columns::
+
+    % mlr join -j account_id -f accounts.dat then group-by account_name balances.dat
+
+Multiple formats including JSON::
+
+    % mlr --json put '$attr = sub($attr, "([0-9]+)_([0-9]+)_.*", "\1:\2")' data/*.json
+
+Aggregate per-column statistics::
+
+    % mlr stats1 -a min,mean,max,p10,p50,p90 -f flag,u,v data/*
+
+Linear regression::
+
+    % mlr stats2 -a linreg-pca -f u,v -g shape data/*
+
+Aggregate custom per-column statistics::
+
+    % mlr put -q '@sum[$a][$b] += $x; end {emit @sum, "a", "b"}' data/*
+
+Iterate over data using DSL expressions::
+
+    % mlr --from estimates.tbl put '
+      for (k,v in $*) {
+        if (is_numeric(v) && k =~ "^[t-z].*$") {
+          $sum += v; $count += 1
+        }
+      }
+      $mean = $sum / $count # no assignment if count unset
+    '
+
+Run DSL expressions from a script file::
+
+    % mlr --from infile.dat put -f analyze.mlr
+
+Split/reduce output to multiple filenames::
+
+    % mlr --from infile.dat put 'tee > "./taps/data-".$a."-".$b, $*'
+
+Compressed I/O::
+
+    % mlr --from infile.dat put 'tee | "gzip > ./taps/data-".$a."-".$b.".gz", $*'
+
+Interoperate with other data-processing tools using standard pipes::
+
+    % mlr --from infile.dat put -q '@v=$*; dump | "jq .[]"'
+
+Tap/trace::
+
+    % mlr --from infile.dat put  '(NR % 1000 == 0) { print > stderr, "Checkpoint ".NR}'

docs/bld

@@ -0,0 +1,6 @@
+poki < about.rst.in      > about.rst
+poki < 10min.rst.in      > 10min.rst
+poki < doc-status.rst.in > doc-status.rst
+poki < cookbook.rst.in   > cookbook.rst
+poki < faq.rst.in        > faq.rst
+poki < reference.rst.in  > reference.rst

docs/cookbook.rst

@@ -0,0 +1,7 @@
+..
+    PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
+
+Cookbook
+=========
+
+To be ported

docs/cookbook.rst.in

@@ -0,0 +1,4 @@
+Cookbook
+=========
+
+To be ported

docs/doc-status.rst

@@ -0,0 +1,7 @@
+..
+    PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
+
+Documentation Status
+======================
+
+As of September 2020 these docs are still being ported from http://johnkerl.org/miller/doc/. Please see https://github.com/johnkerl/miller/issues/375 for more information, including how you can help out.

docs/doc-status.rst.in

@@ -0,0 +1,4 @@
+Documentation Status
+======================
+
+As of September 2020 these docs are still being ported from http://johnkerl.org/miller/doc/. Please see https://github.com/johnkerl/miller/issues/375 for more information, including how you can help out.

docs/faq.rst

@@ -0,0 +1,7 @@
+..
+    PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
+
+FAQ
+=========
+
+To be ported

docs/faq.rst.in

@@ -0,0 +1,4 @@
+FAQ
+=========
+
+To be ported

docs/index.rst

@@ -1,5 +1,5 @@
-Miller Docs v2 (under construction)
-===================================
+Miller Docs v2
+==============
 
 Overview
 ^^^^^^^^
@@ -10,13 +10,27 @@ Overview
 
    about
    10min
+   doc-status
 
 Using Miller
 ^^^^^^^^^^^^
 
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents
+
+   cookbook
+   faq
+
 Reference
 ^^^^^^^^^
 
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents
+
+   reference
+
 Background
 ^^^^^^^^^^
 

docs/poki

@@ -11,6 +11,10 @@ def main
   input_handle = $stdin
   output_handle = $stdout
 
+  output_handle.puts("..")
+  output_handle.puts("    PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.")
+  output_handle.puts("")
+
   input_handle.readlines.each do |content_line|
 
   if content_line =~ /POKI_INCLUDE_ESCAPED\(([^)]+)\)HERE/

docs/reference.rst

@@ -0,0 +1,7 @@
+..
+    PLEASE DO NOT EDIT DIRECTLY. EDIT THE .rst.in FILE PLEASE.
+
+Reference
+=========
+
+To be ported

docs/reference.rst.in

@@ -0,0 +1,4 @@
+Reference
+=========
+
+To be ported

docs/todo.txt

@@ -1,5 +1,6 @@
 * document _static CSS mods
 * do more such
+* document about `make clean html` after editing _static css
 
 * document (after rename) poki-filter and foo.rst.in -> foo.rst
 * implement that in the makefile