README-profiling.md

go/README-profiling.md

@@ -1 +1,79 @@
-# xxx
+# How to run the profiler
+
+Running Miller:
+
+```
+mlr --cpuprofile cpu.pprof put -f u/example.mlr then nothing ~/tmp/huge > /dev/null
+```
+
+(or whatever command-line flags).
+
+# How to view the profiling results
+
+
+## Text mode
+
+```
+go tool pprof mlr cpu.pprof
+top10
+```
+
+## PDF mode
+
+```
+go tool pprof --pdf mlr cpu.pprof > mlr-call-graph.pdf
+mv mlr-call-graph.pdf ~/Desktop
+```
+
+## Flame-graph mode
+
+One-time setup:
+
+
+```
+export GOPATH=$HOME/go
+mkdir -p $HOME/go
+```
+
+```
+go get -u github.com/google/pprof
+ll ~/go/bin/pprof
+go get github.com/uber/go-torch
+```
+
+```
+cd ~/git
+git clone https://github.com/brendangregg/FlameGraph
+```
+
+Per run:
+
+```
+cd /path/to/mlr/go
+export PATH=${PATH}:~/git/FlameGraph/
+go-torch cpu.pprof
+mv torch.svg ~/Desktop/
+```
+
+# How to control garbage collection
+
+```
+# Note 100 is the default
+# Raise the bar for GC threshold:
+GOGC=200  GODEBUG=gctrace=1 mlr -n put -q -f u/mand.mlr 1> /dev/null
+
+# Raise the bar higher for GC threshold:
+GOGC=1000 GODEBUG=gctrace=1 mlr -n put -q -f u/mand.mlr 1> /dev/null
+
+# Turn off GC entirely and see where time is spent:
+GOGC=off  GODEBUG=gctrace=1 mlr -n put -q -f u/mand.mlr 1> /dev/null
+```
+
+# Findings from the profiler as of 2021-02
+
+* GC on: lots of GC
+* GC off: lots of `duffcopy` and `madvise`
+* From benchmark `u/mand.mlr`: issue is allocation created by expressions
+  * Things like `type BinaryFunc func(input1 *Mlrval, input2 *Mlrval) (output Mlrval)`
+  * `z = 2 + x + 4 + y * 3` results in AST, mapped to a CST, with a malloc on the output of every unary/binary/ternary function
+  * Idea: replace with `type BinaryFunc func(output* Mlrval, input1 *Mlrval, input2 *Mlrval)`: no allocation at all.

go/mlr.go

@@ -24,11 +24,7 @@ func main() {
 	// and properly stopping the profile only when main ends via chan-sync,
 	// results in a zero-length pprof file.
 	//
-	// Usage:
-	// * mlr --cpuprofile cpu.pprof put -f u/example.mlr then nothing ~/tmp/huge > /dev/null
-	// * go tool pprof mlr cpu.pprof
-	//   top10
-	// * go tool pprof --pdf mlr cpu.pprof > mlr-call-graph.pdf
+	// Please see README-profiling.md for more information.
 
 	if len(os.Args) >= 3 && os.Args[1] == "--cpuprofile" {
 		profFilename := os.Args[2]
@@ -50,9 +46,9 @@ func main() {
 	}
 
 	//  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-	// 'mlr lecat' or any other non-miller-per-se toolery which is delivered
-	// (for convenience) within the mlr executable. If argv[1] is found then
-	// this function will not return.
+	// 'mlr repl' or 'mlr lecat' or any other non-miller-per-se toolery which
+	// is delivered (for convenience) within the mlr executable. If argv[1] is
+	// found then this function will not return.
 	auxents.Dispatch(os.Args)
 
 	//  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

go/todo.txt

@@ -267,21 +267,6 @@ GOCC UPSTREAMS:
 
 * support "abc" (not just 'a' 'b' 'c') in the lexer part
 
-----------------------------------------------------------------
-FLAME GRAPHS
-
-go get -u github.com/google/pprof
-ll ~/go/bin/pprof
-go get github.com/uber/go-torch
-
-cd ~/git
-git clone https://github.com/brendangregg/FlameGraph
-
-cd /path/to/mlr/go
-export PATH=${PATH}:~/git/FlameGraph/
-go-torch cpu.pprof
-mv torch.svg ~/Desktop/
-
 ----------------------------------------------------------------
 NITS/NON-IMMEDIATE: