Specifically, I did simple experiments in several languages — Ruby, +Python, Lua, Rust, Go, D. In one I just read lines and printed them back out +— a line-oriented cat. In another I consumed input lines like +x=1,y=2,z=3 one at a time, split them on commas and equals signs to +populate hash maps, transformed them (e.g. remove the y field), and +emitted them. Basically mlr cut -x -f y with DKVP format. I +didn’t do anything fancy — just using each language’s +getline, string-split, hashmap-put, etc. And nothing was as fast as +C, so I used C. Here are the experiments I kept (I failed to keep the +Lua code, for example): +C cat, +another C cat, +D cat, +Go cat, +another Go cat, +Rust cat, +Nim cat, +D cut, +Go cut, +Nim cut.
One of Go’s most powerful features is the ease with which it allows quick-to-code, error-free concurrency. Yet Miller, like most high-volume diff --git a/doc/cookbook.html b/doc/cookbook.html index 80cad36c9..a4a48ccc8 100644 --- a/doc/cookbook.html +++ b/doc/cookbook.html @@ -608,33 +608,33 @@ $ mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put ' ' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds i o fcount seconds_delta 1 1 1 0 -2 2 3 0.000033140 -3 3 5 0.000011921 +2 2 3 0.000031948 +3 3 5 0.000013113 4 5 9 0.000015974 5 8 15 0.000020027 -6 13 25 0.000029087 -7 21 41 0.000042915 -8 34 67 0.000066996 -9 55 109 0.000104904 -10 89 177 0.000169039 -11 144 287 0.000321150 -12 233 465 0.000429869 -13 377 753 0.000709057 -14 610 1219 0.001135111 -15 987 1973 0.001828909 -16 1597 3193 0.002882004 -17 2584 5167 0.004646063 -18 4181 8361 0.010217905 -19 6765 13529 0.014159918 -20 10946 21891 0.019574165 -21 17711 35421 0.031366825 -22 28657 57313 0.050403118 -23 46368 92735 0.089492083 -24 75025 150049 0.139133930 -25 121393 242785 0.208625078 -26 196418 392835 0.348189831 -27 317811 635621 0.540225029 -28 514229 1028457 0.867680073 +6 13 25 0.000029802 +7 21 41 0.000044107 +8 34 67 0.000068903 +9 55 109 0.000108004 +10 89 177 0.000172138 +11 144 287 0.000277996 +12 233 465 0.000442028 +13 377 753 0.000705004 +14 610 1219 0.001137972 +15 987 1973 0.001832008 +16 1597 3193 0.003048897 +17 2584 5167 0.004843950 +18 4181 8361 0.009150982 +19 6765 13529 0.015188217 +20 10946 21891 0.022678852 +21 17711 35421 0.036461115 +22 28657 57313 0.052274942 +23 46368 92735 0.091028929 +24 75025 150049 0.138079166 +25 121393 242785 0.235754967 +26 196418 392835 0.368572950 +27 317811 635621 0.592602968 +28 514229 1028457 0.926891088 @@ -666,32 +666,32 @@ $ mlr --ofmt '%.9lf' --opprint seqgen --start 1 --stop 28 then put ' ' then put '$seconds=systime()' then step -a delta -f seconds then cut -x -f seconds i o fcount seconds_delta 1 1 1 0 -2 2 3 0.000043869 -3 3 3 0.000014067 -4 5 3 0.000011921 +2 2 3 0.000035048 +3 3 3 0.000012875 +4 5 3 0.000010967 5 8 3 0.000011206 6 13 3 0.000010967 7 21 3 0.000010014 -8 34 3 0.000010967 +8 34 3 0.000010014 9 55 3 0.000011921 -10 89 3 0.000011921 -11 144 3 0.000011206 -12 233 3 0.000014782 -13 377 3 0.000012159 +10 89 3 0.000010014 +11 144 3 0.000010014 +12 233 3 0.000014067 +13 377 3 0.000010967 14 610 3 0.000010014 -15 987 3 0.000010967 -16 1597 3 0.000010967 -17 2584 3 0.000010014 -18 4181 3 0.000010967 -19 6765 3 0.000010967 +15 987 3 0.000010014 +16 1597 3 0.000010014 +17 2584 3 0.000008821 +18 4181 3 0.000010014 +19 6765 3 0.000010014 20 10946 3 0.000010014 -21 17711 3 0.000010967 +21 17711 3 0.000010014 22 28657 3 0.000010014 -23 46368 3 0.000013113 +23 46368 3 0.000011921 24 75025 3 0.000010967 -25 121393 3 0.000010967 +25 121393 3 0.000010014 26 196418 3 0.000010014 -27 317811 3 0.000010967 +27 317811 3 0.000010014 28 514229 3 0.000010014 diff --git a/doc/why.html b/doc/why.html new file mode 100644 index 000000000..f4ba64ee0 --- /dev/null +++ b/doc/why.html @@ -0,0 +1,326 @@ + + + + + + + + + + + +|
+
+
+
+
+ Overview: + • About Miller + • Miller in 10 minutes + • File formats + • Miller features in the context of the Unix toolkit + • Record-heterogeneity + • Internationalization + Using Miller: + • FAQ + • Data-diving examples + • Cookbook + • Manpage + • Reference + • Reference: Verbs + • Reference: DSL + • Documents by release + • Installation, portability, dependencies, and testing + Background: + • Why? + • Why C? + • Why call it Miller? + • How original is Miller? + • Performance + Repository: + • Things to do + • Contact information + • GitHub repo + + + + |
+
+
+
+
+
+
+
+
+ Who is Miller for?+ + For background, I’m a software engineer, with a heavy devops bent +and a non-trivial amount of data-engineering in my career. +What was Miller created to do?+ + First: there are tools like xsv which handles CSV marvelously and +jq which handles JSON marvelously, and so on — but I over the +years of my career in the software industry I’ve found myself, and +others, doing a lot of ad-hoc things which really were fundamentally the same +except for format. So the number one thing about Miller is doing common +things while supportingTradeoffs+ + Miller is command-line-only by design. People who want a graphical user +interface won’t find it here. This is in part (a) accommodating my +personal preferences, and in part (b) guided by my experience/belief that the +command line is very expressive. Steep learning curve, yes. I consider that +price worth paying. + + Another tradeoff: supporting lists of records — each with only one +depth — keeps me supporting only what can be expressed in all of +those formats. E.g. in JSON you can have lists of lists of lists which Miller +just doesn’t handle. So Miller can’t (and won’t) handle +arbitrary JSON because it only handles tabular data which can be expressed in a +variety of formats. + + A third tradeoff is doing build-from-scratch in a low-level language. +It’d be quicker to write (but slower to run) if written in a high-level +language. If Miller were written in Python, it would be implemented in +significantly fewer lines of code than its current C implementation. The DSL +would just be an eval of Python code. And it would run slower, but +maybe not enough slower to be a problem for most folks. + + A fourth tradeoff is in the DSL (more visibly so in 5.0.0 but already in +pre-5.0.0): how much to make it dynamically typed — so you can just say +y=x+1 with a minimum number of keystrokes — vs. having it do a good job +of telling you when you’ve made a typo. This is a common paradigm across +all languages. Some like Ruby you don’t declare anything and +they’re quick to code little stuff in but programs of even a few thousand +lines (which isn’t large in the software world) become insanely +unmanageable. Then Java at the other extreme which is very typesafe but you +have to type in a lot of punctuation, angle brackets, datatypes, repetition, +etc. just to be able to get anything done. And some in the middle like Go which +are typesafe but with type inference which aim to do the best of both. In the +Miller (5.0.0) DSL you get y=x+1 by default but you can have things like int y += x+1 etc. so the typesafety is opt-in. See also here for more information on +type-checking. + +Moving forward+ + I originally aimed Miller at people who already know what +sed/awk/cut/sort/join are and +wanted some options. But as time goes by I realize that tools like this can be +useful to folks who don’t know what those things are; people who +aren’t primarily coders; people who are scientists, or data scientists. +These days some journalists do data analysis. So moving forward in terms of +docs, I am working on having more cookbook, follow-by-example stuff in addition +to the existing language-reference kinds of stuff. And prioritizing a Windows +port — which is way overdue. And continuing to seek out input from people +who use Miller on where to go next. + |
+
+
Specifically, I did simple experiments in several languages — Ruby, +Python, Lua, Rust, Go, D. In one I just read lines and printed them back out +— a line-oriented cat. In another I consumed input lines like +x=1,y=2,z=3 one at a time, split them on commas and equals signs to +populate hash maps, transformed them (e.g. remove the y field), and +emitted them. Basically mlr cut -x -f y with DKVP format. I +didn’t do anything fancy — just using each language’s +getline, string-split, hashmap-put, etc. And nothing was as fast as +C, so I used C. Here are the experiments I kept (I failed to keep the +Lua code, for example): +C cat, +another C cat, +D cat, +Go cat, +another Go cat, +Rust cat, +Nim cat, +D cut, +Go cut, +Nim cut.
One of Go’s most powerful features is the ease with which it allows quick-to-code, error-free concurrency. Yet Miller, like most high-volume diff --git a/perf/catc.c.txt b/perf/catc.c.txt new file mode 100644 index 000000000..6401222a3 --- /dev/null +++ b/perf/catc.c.txt @@ -0,0 +1,45 @@ +#include