diff --git a/docs6/docs/cpu.md b/docs6/docs/cpu.md new file mode 100644 index 000000000..3934cebe1 --- /dev/null +++ b/docs6/docs/cpu.md @@ -0,0 +1,44 @@ + +
+ +Quick links: +  +Verb list +  +Function list +  +Glossary +  +Repository ↗ + +
+# CPU/multicore usage + +Miller 6 is written in [Go](https://golang.org/) which supports multicore programming. + +Miller uses Go's _channel_ concept. The following are all separate goroutines: + +* One channel for input-record reader: parsing input file(s) to record objects +* One channel for each [verb](reference-verbs.md) in [then-chains](reference-main-then-chaining.md) +* One channel for output-record writer: formatting output records as text +* One controller channel which coordinates all these, without much work to do. + +For example, `mlr --csv cut -f somefield then sort -f otherfield then put '$z = +$x + $y' a.csv b.csv c.csv` will have 6 goroutines running: input-reader, +`cut`, `sort`, `put`, output-writer, controller. + +If all the verbs in the chain are [streaming](streaming-and-memory.md) -- +operating on each record as it arrives, then passing it on -- then all verbs in +the chain will be active at once. On the other hand, if there is a +non-streaming verb in the chain, which produces output only after receiving all +input -- for example, `sort` -- then we would expect verbs after that in the +chain to sit idle until the end of the input stream is reached, the `sort` does +its computation, then sends its output to downstream verbs. + +In practice, profiling has shown that the input-reader uses the most CPU of all +the above. This means CPUs running verbs may not be 100% utilized, since they +are likely to be spending some of their time waiting for input data. + +You can set the Go-standard environment variable `GOMAXPROCS` if you like. If +you don't, Miller will (as is standard for Go programs in Go 1.16 and above) up +to all available CPUs. diff --git a/docs6/docs/cpu.md.in b/docs6/docs/cpu.md.in new file mode 100644 index 000000000..520cd54b0 --- /dev/null +++ b/docs6/docs/cpu.md.in @@ -0,0 +1,30 @@ +# CPU/multicore usage + +Miller 6 is written in [Go](https://golang.org/) which supports multicore programming. + +Miller uses Go's _channel_ concept. The following are all separate goroutines: + +* One channel for input-record reader: parsing input file(s) to record objects +* One channel for each [verb](reference-verbs.md) in [then-chains](reference-main-then-chaining.md) +* One channel for output-record writer: formatting output records as text +* One controller channel which coordinates all these, without much work to do. + +For example, `mlr --csv cut -f somefield then sort -f otherfield then put '$z = +$x + $y' a.csv b.csv c.csv` will have 6 goroutines running: input-reader, +`cut`, `sort`, `put`, output-writer, controller. + +If all the verbs in the chain are [streaming](streaming-and-memory.md) -- +operating on each record as it arrives, then passing it on -- then all verbs in +the chain will be active at once. On the other hand, if there is a +non-streaming verb in the chain, which produces output only after receiving all +input -- for example, `sort` -- then we would expect verbs after that in the +chain to sit idle until the end of the input stream is reached, the `sort` does +its computation, then sends its output to downstream verbs. + +In practice, profiling has shown that the input-reader uses the most CPU of all +the above. This means CPUs running verbs may not be 100% utilized, since they +are likely to be spending some of their time waiting for input data. + +You can set the Go-standard environment variable `GOMAXPROCS` if you like. If +you don't, Miller will (as is standard for Go programs in Go 1.16 and above) up +to all available CPUs. diff --git a/docs6/docs/installation.md b/docs6/docs/installation.md index a8d600a37..0d65f7eb8 100644 --- a/docs6/docs/installation.md +++ b/docs6/docs/installation.md @@ -33,6 +33,8 @@ Note: sudo port selfupdate && sudo port install miller +Note that Homebrew is available for Linux as well: [https://docs.brew.sh/linux](https://docs.brew.sh/linux). + You may already have the `mlr` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following: 
diff --git a/docs6/docs/installation.md.in b/docs6/docs/installation.md.in
index 6ec720c8d..1fdf5435b 100644
--- a/docs6/docs/installation.md.in
+++ b/docs6/docs/installation.md.in
@@ -19,6 +19,8 @@ GENMD_CARDIFY_HIGHLIGHT_ONE
 sudo port selfupdate && sudo port install miller
 GENMD_EOF
 
+Note that Homebrew is available for Linux as well: [https://docs.brew.sh/linux](https://docs.brew.sh/linux).
+
 You may already have the `mlr` executable available in your platform's package manager on NetBSD, Debian Linux, Ubuntu Xenial and upward, Arch Linux, or perhaps other distributions. For example, on various Linux distributions you might do one of the following:
 
 GENMD_CARDIFY_HIGHLIGHT_ONE
diff --git a/docs6/docs/installing-miller.md b/docs6/docs/installing-miller.md
index 8e72275fe..435093174 100644
--- a/docs6/docs/installing-miller.md
+++ b/docs6/docs/installing-miller.md
@@ -20,7 +20,7 @@ You can install Miller for various platforms as follows.
     * You can get latest Miller 6 builds for Linux, MacOS, and Windows by visiting [https://github.com/johnkerl/miller/actions](https://github.com/johnkerl/miller/actions), selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.)
     * See also the [build page](build.md) if you prefer -- in particular, if your platform's package manager doesn't have the latest release.
 * Miller 5 is released, and is described by [https://miller.readthedocs.io](https://miller.readthedocs.io).
-    * Linux: `yum install miller` or `apt-get install miller` depending on your flavor of Linux.
+    * Linux: `yum install miller` or `apt-get install miller` depending on your flavor of Linux, or [Homebrew](https://docs.brew.sh/linux).
     * MacOS: `brew install miller` or `port install miller` depending on your preference of [Homebrew](https://brew.sh) or [MacPorts](https://macports.org).
     * Windows: `choco install miller` using [Chocolatey](https://chocolatey.org).
 
diff --git a/docs6/docs/installing-miller.md.in b/docs6/docs/installing-miller.md.in
index c656a179d..2899ddcd5 100644
--- a/docs6/docs/installing-miller.md.in
+++ b/docs6/docs/installing-miller.md.in
@@ -6,7 +6,7 @@ You can install Miller for various platforms as follows.
     * You can get latest Miller 6 builds for Linux, MacOS, and Windows by visiting [https://github.com/johnkerl/miller/actions](https://github.com/johnkerl/miller/actions), selecting the latest build, and clicking _Artifacts_. (These are retained for 5 days after each commit.)
     * See also the [build page](build.md) if you prefer -- in particular, if your platform's package manager doesn't have the latest release.
 * Miller 5 is released, and is described by [https://miller.readthedocs.io](https://miller.readthedocs.io).
-    * Linux: `yum install miller` or `apt-get install miller` depending on your flavor of Linux.
+    * Linux: `yum install miller` or `apt-get install miller` depending on your flavor of Linux, or [Homebrew](https://docs.brew.sh/linux).
     * MacOS: `brew install miller` or `port install miller` depending on your preference of [Homebrew](https://brew.sh) or [MacPorts](https://macports.org).
     * Windows: `choco install miller` using [Chocolatey](https://chocolatey.org).
 
diff --git a/docs6/docs/proofreads.txt b/docs6/docs/proofreads.txt
index 5ac200f47..d102f7f6f 100644
--- a/docs6/docs/proofreads.txt
+++ b/docs6/docs/proofreads.txt
@@ -1,6 +1,13 @@
-----------------------------------------------------------------
-! merge 3 new 4 !
+TOP:
 
+E data-types page
+E flatten/unflatten page
+c! repifs !!
+c! seps \001 etc !
+C flags LUTs
+? twi-dm re all-contribs: all-contributors.org
+
+----------------------------------------------------------------
 ALL:
 * csv to csv,tsv throughout
 * rid of explicitly passing around os.Stdout in all various help functions, annoying
@@ -19,14 +26,18 @@ ALL:
   https://squidfunk.github.io/mkdocs-material/customization/#extending-the-theme
 
 ----------------------------------------------------------------
-c! repifs !!
-c! seps \001 etc !
-C flags LUTs
-? twi-dm re all-contribs: all-contributors.org
 
 e fzf-ish w/ head -n 4, --from, up-arrow & append verb, then cat -- find & update the existing section
 
+? figure out flatsep vs iflatsep/oflatsep & have a story around the choice either way
+  - ctx ptr-split for invars
 E fill out flatten/unflatten page
+  echo a.b.c=3 | mlr --ojson cat
+  echo a.b.c=3 | mlr --ojson --no-jvstack cat | mlr --j2c --oflatsep : cat
+  echo a:b:c=3 | mlr --ojson --no-jvstack cat
+  echo a:b:c=3 | mlr --ojson --no-jvstack --oflatsep : cat
+  -> call it --flatsep
+
 E fill out data-types page
 E fill out arrays page
   - xref to fla/unfla page
@@ -37,11 +48,6 @@ E fill out maps page
 E make flags page
   C show-all-flags help -- fully alpha, or alpha-by-type; LUT refactor
 
-c GOMAXPROCS -- up it? separate page maybe -- ?
-  - note one goroutine for in, out, & each verb
-  - check and respect env-var
-  - add note on threading and how it scales on multicore arches
-
 * functions: somewhere organize by type. olh/man6/docs6 ...
 
 * regex: more about what is / is not
@@ -154,12 +160,8 @@ E Keep in mind that out-of-stream variables are a nested, multi-level hashmap (d
   o why not '--oflatsep /' respected?
 
 diffs other langs:
-e slice incl/excl; split-in-half
+e slice incl/excl; split-in-half: x[:2] x[2:] vs x[:2] x[3:]
 e localvars in if/else
-e elif
-
-glossary:
-! after it's done, link-check *all* ........ :^/
 
 separators:
 E write up which file formats support which flags
@@ -167,7 +169,4 @@ E write up which file formats support which flags
 manpage:
 ? [NEEDS READ-THROUGH]
 
-install:
-* n.b. homebrew linux https://github.com/johnkerl/miller/issues/77#issuecomment-536611087
-
 mk-func-table.rb: port comments from sphinx to mkdocs
diff --git a/docs6/docs/reference-dsl-differences.md b/docs6/docs/reference-dsl-differences.md
index 1dbc6627d..2d583bbd4 100644
--- a/docs6/docs/reference-dsl-differences.md
+++ b/docs6/docs/reference-dsl-differences.md
@@ -69,6 +69,10 @@ Expected one of:
   * / // % .* ./ .// ** [ [[ [[[
+## elif + +Miller has [`elif`](reference-dsl-control-structures.md#if-statements), not `else if` or `elsif`. + ## Required curly braces Bodies for all compound statements must be enclosed in curly braces, even if the body is a single statement: diff --git a/docs6/docs/reference-dsl-differences.md.in b/docs6/docs/reference-dsl-differences.md.in index 7632652fa..0d5f5bc12 100644 --- a/docs6/docs/reference-dsl-differences.md.in +++ b/docs6/docs/reference-dsl-differences.md.in @@ -47,6 +47,10 @@ mlr --csv --from example.csv put -q ' ' GENMD_EOF +## elif + +Miller has [`elif`](reference-dsl-control-structures.md#if-statements), not `else if` or `elsif`. + ## Required curly braces Bodies for all compound statements must be enclosed in curly braces, even if the body is a single statement: diff --git a/docs6/docs/reference-verbs.md b/docs6/docs/reference-verbs.md index 388f4f520..6734d0fcb 100644 --- a/docs6/docs/reference-verbs.md +++ b/docs6/docs/reference-verbs.md @@ -2072,7 +2072,8 @@ Please see the [DSL reference](reference-dsl.md) for more information about the 
 Usage: mlr regularize [options]
-Outputs records sorted lexically ascending by keys.Options:
+Outputs records sorted lexically ascending by keys.
+Options:
 -h|--help Show this message.
diff --git a/docs6/mkdocs.yml b/docs6/mkdocs.yml index 003aa0eaf..ccab3e8ac 100644 --- a/docs6/mkdocs.yml +++ b/docs6/mkdocs.yml @@ -63,6 +63,7 @@ nav: - "Separators": "reference-main-separators.md" - "List of verbs": "reference-verbs.md" - "Streaming processing, and memory usage": "streaming-and-memory.md" + - "CPU/multicore usage": "cpu.md" - "Data types": "reference-main-data-types.md" - "Arithmetic": "reference-main-arithmetic.md" - "Maps": "reference-main-maps.md" diff --git a/go/mlr.go b/go/mlr.go index 93686c671..9b2894bd4 100644 --- a/go/mlr.go +++ b/go/mlr.go @@ -6,13 +6,30 @@ import ( "runtime" "runtime/debug" "runtime/pprof" + "strconv" "mlr/src/entrypoint" ) // ---------------------------------------------------------------- func main() { - runtime.GOMAXPROCS(4) // Seems reasonable these days + + // Respect env $GOMAXPROCS, if provided, else set default. + haveSetGoMaxProcs := false + goMaxProcsString := os.Getenv("GOMAXPROCS") + if goMaxProcsString != "" { + goMaxProcs, err := strconv.Atoi(goMaxProcsString) + if err != nil { + runtime.GOMAXPROCS(goMaxProcs) + haveSetGoMaxProcs = true + } + } + if !haveSetGoMaxProcs { + // As of Go 1.16 this is the default anyway. For 1.15 and below we need + // to explicitly set this. + runtime.GOMAXPROCS(runtime.NumCPU()) + } + debug.SetGCPercent(500) // Empirical: See README-profiling.md // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/go/todo.txt b/go/todo.txt index d9ab1008e..84a5fab54 100644 --- a/go/todo.txt +++ b/go/todo.txt @@ -1,39 +1,23 @@ ================================================================ TOP OF LIST: -* pre-release: - o explicit punchdown list - o draft release w/ m6 binaries YYYYMMDD -- ? - o gha download steps clearer - ! allcontribs -* doc6 proofreads +! doc6 proofreads.txt * 'explicit r' issue 297: code, UT, doc * shell-commands.html; while-read -* memory/streaming page +* repifs * "\001" et al. * * I/O redirects * fmtnum * localtime * stats1 -r/--fr * * --nr-progress-mod * mlr -k * json comment-handling * * cli-audits * doc6 * survey * check issues * regtest pendings * * survey followups/discussion-post * -* PD wtf ! - -* GOMAXPROCS env-override, else default 8 with comment - ----------------------------------------------------------------- -* mlr repl (w/o mrpl) doesn't print final newline on EOF -* also: feature/shorthand for repl newline before prompt - -* slice-indexing bug: - - μεταμόρφωσις: x=[1,2,3,4,5] - μεταμόρφωσις: x[:2] - [1, 2] - μεταμόρφωσις: x[2:] - [2, 3, 4, 5] +d pre-release: + o explicit punchdown list + o draft release w/ m6 binaries YYYYMMDD -- ? + o gha download steps clearer ---------------------------------------------------------------- implicit r: @@ -47,10 +31,8 @@ docs: * mlr --j2c --no-auto-flatten cat $mlg/regtest/input/flatten-input-2.json - code: this is ok ... maybe prefer single-line -- ? -* src/man/doc hygiene @ build script - -w discussion re docs6 ... * single cheatsheet page -- put out RFH? + https://twitter.com/icymi_py/status/1426622817785765898/photo/1 * note somewhere (maybe 'what's new in miller6' and/or to-be-written survey review): o miller was first of all a tool for myself, and anyone else who might find it useful @@ -71,46 +53,19 @@ w discussion re docs6 ... * link to SE table ... https://github.com/johnkerl/miller/discussions/609#discussioncomment-1115715 -* csv -> csv/tsv throughout - -* cheatsheet someday? example: - https://twitter.com/icymi_py/status/1426622817785765898/photo/1 - ----------------------------------------------------------------- -memory/streaming page - -* https://github.com/johnkerl/miller/issues/587 - ---------------------------------------------------------------- DOC6 * colored-shapes.dkvp -> csv; also mkdat2 * data/small -> csv throughout. and/or just use example.csv -* quicklinks -- ? -* vim syntax highlighting for .mlr files -- highlight this somewhere * shell-commands.html while-read example issue -* memory page: - o open with "out of memory" -- what next? - o streaming vs non-streaming - o unsparsify -f - o no out-of-core sort yet ... - o older versions - * number-formatting page -* list-of-verbs shortlist @ top - * check <-> manpage/olh sections & doc6 sections -* fill out: - o new-in-miller-6: flatten/unflatten - o JSON flatten/unflatten section - o Arrays section including 1-up and why - o differences from other languages (cf. juliadocs) - * doc notes about --jknquotein & --jvquoteall. make sure they're ignore-accepted in the CLIP. * file-formats: @@ -168,6 +123,9 @@ BLOCKERS μεταμόρφωσις: z[1].x + z[2].x (absent) +! repifs +! seps "\001" etc + * finish stats1 -r * what about -I mixed with .gzin -- ? @@ -226,6 +184,8 @@ NON-BLOCKERS * non-blocker: commenting passes ... +* non-blocker: feature/shorthand for repl newline before prompt + * non-blocker: new functions: o new columns-to-arrays and arrays-to-columns for stan format