misc to-do items

docs6/docs/cpu.md

@@ -0,0 +1,44 @@
+<!---  PLEASE DO NOT EDIT DIRECTLY. EDIT THE .md.in FILE PLEASE. --->
+<div>
+<span class="quicklinks">
+Quick links:
+&nbsp;
+<a class="quicklink" href="../reference-verbs/index.html">Verb list</a>
+&nbsp;
+<a class="quicklink" href="../reference-dsl-builtin-functions/index.html">Function list</a>
+&nbsp;
+<a class="quicklink" href="../glossary/index.html">Glossary</a>
+&nbsp;
+<a class="quicklink" href="https://github.com/johnkerl/miller" target="_blank">Repository ↗</a>
+</span>
+</div>
+# CPU/multicore usage
+
+Miller 6 is written in [Go](https://golang.org/) which supports multicore programming.
+
+Miller uses Go's _channel_ concept. The following are all separate goroutines:
+
+* One channel for input-record reader: parsing input file(s) to record objects
+* One channel for each [verb](reference-verbs.md) in [then-chains](reference-main-then-chaining.md)
+* One channel for output-record writer: formatting output records as text
+* One controller channel which coordinates all these, without much work to do.
+
+For example, `mlr --csv cut -f somefield then sort -f otherfield then put '$z =
+$x + $y' a.csv b.csv c.csv` will have 6 goroutines running: input-reader,
+`cut`, `sort`, `put`, output-writer, controller.
+
+If all the verbs in the chain are [streaming](streaming-and-memory.md) --
+operating on each record as it arrives, then passing it on -- then all verbs in
+the chain will be active at once. On the other hand, if there is a
+non-streaming verb in the chain, which produces output only after receiving all
+input -- for example, `sort` -- then we would expect verbs after that in the
+chain to sit idle until the end of the input stream is reached, the `sort` does
+its computation, then sends its output to downstream verbs.
+
+In practice, profiling has shown that the input-reader uses the most CPU of all
+the above. This means CPUs running verbs may not be 100% utilized, since they
+are likely to be spending some of their time waiting for input data.
+
+You can set the Go-standard environment variable `GOMAXPROCS` if you like. If
+you don't, Miller will (as is standard for Go programs in Go 1.16 and above) up
+to all available CPUs.

docs6/docs/cpu.md.in

@@ -0,0 +1,30 @@
+# CPU/multicore usage
+
+Miller 6 is written in [Go](https://golang.org/) which supports multicore programming.
+
+Miller uses Go's _channel_ concept. The following are all separate goroutines:
+
+* One channel for input-record reader: parsing input file(s) to record objects
+* One channel for each [verb](reference-verbs.md) in [then-chains](reference-main-then-chaining.md)
+* One channel for output-record writer: formatting output records as text
+* One controller channel which coordinates all these, without much work to do.
+
+For example, `mlr --csv cut -f somefield then sort -f otherfield then put '$z =
+$x + $y' a.csv b.csv c.csv` will have 6 goroutines running: input-reader,
+`cut`, `sort`, `put`, output-writer, controller.
+
+If all the verbs in the chain are [streaming](streaming-and-memory.md) --
+operating on each record as it arrives, then passing it on -- then all verbs in
+the chain will be active at once. On the other hand, if there is a
+non-streaming verb in the chain, which produces output only after receiving all
+input -- for example, `sort` -- then we would expect verbs after that in the
+chain to sit idle until the end of the input stream is reached, the `sort` does
+its computation, then sends its output to downstream verbs.
+
+In practice, profiling has shown that the input-reader uses the most CPU of all
+the above. This means CPUs running verbs may not be 100% utilized, since they
+are likely to be spending some of their time waiting for input data.
+
+You can set the Go-standard environment variable `GOMAXPROCS` if you like. If
+you don't, Miller will (as is standard for Go programs in Go 1.16 and above) up
+to all available CPUs.

docs6/docs/installation.md

@@ -33,6 +33,8 @@ Note:
 <b>sudo port selfupdate && sudo port install miller</b>
 </pre>
 
+Note that Homebrew is available for Linux as well: [https://docs.brew.sh/linux](https://docs.brew.sh/linux).
+
 You may already have the `mlr` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following:
 
 <pre class="pre-highlight-non-pair">

docs6/docs/installation.md.in

@@ -19,6 +19,8 @@ GENMD_CARDIFY_HIGHLIGHT_ONE
 sudo port selfupdate && sudo port install miller
 GENMD_EOF
 
+Note that Homebrew is available for Linux as well: [https://docs.brew.sh/linux](https://docs.brew.sh/linux).
+
 You may already have the `mlr` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following:
 
 GENMD_CARDIFY_HIGHLIGHT_ONE

docs6/docs/installing-miller.md

@@ -20,7 +20,7 @@ You can install Miller for various platforms as follows.
     * You can get latest Miller 6 builds for Linux, MacOS, and Windows by visiting [https://github.com/johnkerl/miller/actions](https://github.com/johnkerl/miller/actions), selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.)
     * See also the [build page](build.md) if you prefer -- in particular, if your platform's package manager doesn't have the latest release.
 * Miller 5 is released, and is described by [https://miller.readthedocs.io](https://miller.readthedocs.io).
-    * Linux: `yum install miller` or `apt-get install miller` depending on your flavor of Linux.
+    * Linux: `yum install miller` or `apt-get install miller` depending on your flavor of Linux, or [Homebrew](https://docs.brew.sh/linux).
     * MacOS: `brew install miller` or `port install miller` depending on your preference of [Homebrew](https://brew.sh) or [MacPorts](https://macports.org).
     * Windows: `choco install miller` using [Chocolatey](https://chocolatey.org).
 

docs6/docs/installing-miller.md.in

@@ -6,7 +6,7 @@ You can install Miller for various platforms as follows.
     * You can get latest Miller 6 builds for Linux, MacOS, and Windows by visiting [https://github.com/johnkerl/miller/actions](https://github.com/johnkerl/miller/actions), selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.)
     * See also the [build page](build.md) if you prefer -- in particular, if your platform's package manager doesn't have the latest release.
 * Miller 5 is released, and is described by [https://miller.readthedocs.io](https://miller.readthedocs.io).
-    * Linux: `yum install miller` or `apt-get install miller` depending on your flavor of Linux.
+    * Linux: `yum install miller` or `apt-get install miller` depending on your flavor of Linux, or [Homebrew](https://docs.brew.sh/linux).
     * MacOS: `brew install miller` or `port install miller` depending on your preference of [Homebrew](https://brew.sh) or [MacPorts](https://macports.org).
     * Windows: `choco install miller` using [Chocolatey](https://chocolatey.org).
 

docs6/docs/proofreads.txt

@@ -1,6 +1,13 @@
-----------------------------------------------------------------
-! merge 3 new 4 !
+TOP:
 
+E data-types page
+E flatten/unflatten page
+c! repifs !!
+c! seps \001 etc !
+C flags LUTs
+? twi-dm re all-contribs: all-contributors.org
+
+----------------------------------------------------------------
 ALL:
 * csv to csv,tsv throughout
 * rid of explicitly passing around os.Stdout in all various help functions, annoying
@@ -19,14 +26,18 @@ ALL:
   https://squidfunk.github.io/mkdocs-material/customization/#extending-the-theme
 
 ----------------------------------------------------------------
-c! repifs !!
-c! seps \001 etc !
-C flags LUTs
-? twi-dm re all-contribs: all-contributors.org
 
 e fzf-ish w/ head -n 4, --from, up-arrow & append verb, then cat -- find & update the existing section
 
+? figure out flatsep vs iflatsep/oflatsep & have a story around the choice either way
+  - ctx ptr-split for invars
 E fill out flatten/unflatten page
+  echo a.b.c=3 | mlr --ojson cat
+  echo a.b.c=3 | mlr --ojson --no-jvstack cat | mlr --j2c --oflatsep : cat
+  echo a:b:c=3 | mlr --ojson --no-jvstack cat
+  echo a:b:c=3 | mlr --ojson --no-jvstack --oflatsep : cat
+  -> call it --flatsep
+
 E fill out data-types page
 E fill out arrays page
   - xref to fla/unfla page
@@ -37,11 +48,6 @@ E fill out maps page
 E make flags page
   C show-all-flags help -- fully alpha, or alpha-by-type; LUT refactor
 
-c GOMAXPROCS -- up it? separate page maybe -- ?
-  - note one goroutine for in, out, & each verb
-  - check and respect env-var
-  - add note on threading and how it scales on multicore arches
-
 * functions: somewhere organize by type. olh/man6/docs6 ...
 
 * regex: more about what is / is not
@@ -154,12 +160,8 @@ E Keep in mind that out-of-stream variables are a nested, multi-level hashmap (d
   o why not '--oflatsep /' respected?
 
 diffs other langs:
-e slice incl/excl; split-in-half
+e slice incl/excl; split-in-half: x[:2] x[2:] vs x[:2] x[3:]
 e localvars in if/else
-e elif
-
-glossary:
-! after it's done, link-check *all* ........ :^/
 
 separators:
 E write up which file formats support which flags
@@ -167,7 +169,4 @@ E write up which file formats support which flags
 manpage:
 ? [NEEDS READ-THROUGH]
 
-install:
-* n.b. homebrew linux https://github.com/johnkerl/miller/issues/77#issuecomment-536611087
-
 mk-func-table.rb: port comments from sphinx to mkdocs

docs6/docs/reference-dsl-differences.md

@@ -69,6 +69,10 @@ Expected one of:
   * / // % .* ./ .// ** [ [[ [[[
 </pre>
 
+## elif
+
+Miller has [`elif`](reference-dsl-control-structures.md#if-statements), not `else if` or `elsif`.
+
 ## Required curly braces
 
 Bodies for all compound statements must be enclosed in curly braces, even if the body is a single statement:

docs6/docs/reference-dsl-differences.md.in

@@ -47,6 +47,10 @@ mlr --csv --from example.csv put -q '
 '
 GENMD_EOF
 
+## elif
+
+Miller has [`elif`](reference-dsl-control-structures.md#if-statements), not `else if` or `elsif`.
+
 ## Required curly braces
 
 Bodies for all compound statements must be enclosed in curly braces, even if the body is a single statement:

docs6/docs/reference-verbs.md

@@ -2072,7 +2072,8 @@ Please see the [DSL reference](reference-dsl.md) for more information about the
 </pre>
 <pre class="pre-non-highlight-in-pair">
 Usage: mlr regularize [options]
-Outputs records sorted lexically ascending by keys.Options:
+Outputs records sorted lexically ascending by keys.
+Options:
 -h|--help Show this message.
 </pre>
 

docs6/mkdocs.yml

@@ -63,6 +63,7 @@ nav:
       - "Separators": "reference-main-separators.md"
       - "List of verbs": "reference-verbs.md"
       - "Streaming processing, and memory usage": "streaming-and-memory.md"
+      - "CPU/multicore usage": "cpu.md"
       - "Data types": "reference-main-data-types.md"
       - "Arithmetic": "reference-main-arithmetic.md"
       - "Maps": "reference-main-maps.md"

go/mlr.go

@@ -6,13 +6,30 @@ import (
 	"runtime"
 	"runtime/debug"
 	"runtime/pprof"
+	"strconv"
 
 	"mlr/src/entrypoint"
 )
 
 // ----------------------------------------------------------------
 func main() {
-	runtime.GOMAXPROCS(4)   // Seems reasonable these days
+
+	// Respect env $GOMAXPROCS, if provided, else set default.
+	haveSetGoMaxProcs := false
+	goMaxProcsString := os.Getenv("GOMAXPROCS")
+	if goMaxProcsString != "" {
+		goMaxProcs, err := strconv.Atoi(goMaxProcsString)
+		if err != nil {
+			runtime.GOMAXPROCS(goMaxProcs)
+			haveSetGoMaxProcs = true
+		}
+	}
+	if !haveSetGoMaxProcs {
+		// As of Go 1.16 this is the default anyway. For 1.15 and below we need
+		// to explicitly set this.
+		runtime.GOMAXPROCS(runtime.NumCPU())
+	}
+
 	debug.SetGCPercent(500) // Empirical: See README-profiling.md
 
 	//  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

go/todo.txt

@@ -1,39 +1,23 @@
 ================================================================
 TOP OF LIST:
 
-* pre-release:
-  o explicit punchdown list
-  o draft release w/ m6 binaries YYYYMMDD -- ?
-  o gha download steps clearer
-  
 ! allcontribs
 
-* doc6 proofreads
+! doc6 proofreads.txt
 * 'explicit r' issue 297: code, UT, doc
 
 * shell-commands.html; while-read
-* memory/streaming page
 
+* repifs * "\001" et al. *
 * I/O redirects * fmtnum * localtime * stats1 -r/--fr *
 * --nr-progress-mod * mlr -k * json comment-handling *
 * cli-audits * doc6 * survey * check issues * regtest pendings *
 * survey followups/discussion-post *
 
-* PD wtf !
-
-* GOMAXPROCS env-override, else default 8 with comment
-
-----------------------------------------------------------------
-* mlr repl (w/o mrpl) doesn't print final newline on EOF
-* also: feature/shorthand for repl newline before prompt
-
-* slice-indexing bug:
-
-  μεταμόρφωσις: x=[1,2,3,4,5]
-  μεταμόρφωσις: x[:2]
-  [1, 2]
-  μεταμόρφωσις: x[2:]
-  [2, 3, 4, 5]
+d pre-release:
+  o explicit punchdown list
+  o draft release w/ m6 binaries YYYYMMDD -- ?
+  o gha download steps clearer
 
 ----------------------------------------------------------------
 implicit r:
@@ -47,10 +31,8 @@ docs:
 * mlr --j2c --no-auto-flatten cat $mlg/regtest/input/flatten-input-2.json
   - code: this is ok ... maybe prefer single-line -- ?
 
-* src/man/doc hygiene @ build script
-
-w discussion re docs6 ...
 * single cheatsheet page -- put out RFH?
+  https://twitter.com/icymi_py/status/1426622817785765898/photo/1
 
 * note somewhere (maybe 'what's new in miller6' and/or to-be-written survey review):
   o miller was first of all a tool for myself, and anyone else who might find it useful
@@ -71,46 +53,19 @@ w discussion re docs6 ...
 * link to SE table ...
   https://github.com/johnkerl/miller/discussions/609#discussioncomment-1115715
 
-* csv -> csv/tsv throughout
-
-* cheatsheet someday? example:
-  https://twitter.com/icymi_py/status/1426622817785765898/photo/1
-
-----------------------------------------------------------------
-memory/streaming page
-
-* https://github.com/johnkerl/miller/issues/587
-
 ----------------------------------------------------------------
 DOC6
 
 * colored-shapes.dkvp -> csv; also mkdat2
 * data/small -> csv throughout. and/or just use example.csv
-* quicklinks -- ?
-* vim syntax highlighting for .mlr files -- highlight this somewhere
 
 * shell-commands.html
   while-read example issue
 
-* memory page:
-  o open with "out of memory" -- what next?
-  o streaming vs non-streaming
-  o unsparsify -f
-  o no out-of-core sort yet ...
-  o older versions
-
 * number-formatting page
 
-* list-of-verbs shortlist @ top
-
 * check <-> manpage/olh sections & doc6 sections
 
-* fill out:
-  o new-in-miller-6: flatten/unflatten
-  o JSON flatten/unflatten section
-  o Arrays section including 1-up and why
-  o differences from other languages (cf. juliadocs)
-
 * doc notes about --jknquotein & --jvquoteall. make sure they're ignore-accepted in the CLIP.
 
 * file-formats:
@@ -168,6 +123,9 @@ BLOCKERS
   μεταμόρφωσις: z[1].x + z[2].x
   (absent)
 
+! repifs
+! seps "\001" etc
+
 * finish stats1 -r
 
 * what about -I mixed with .gzin -- ?
@@ -226,6 +184,8 @@ NON-BLOCKERS
 
 * non-blocker: commenting passes ...
 
+* non-blocker: feature/shorthand for repl newline before prompt
+
 * non-blocker: new functions:
   o new columns-to-arrays and arrays-to-columns for stan format