mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-19 09:24:14 +00:00
116 lines
5.2 KiB
HTML
116 lines
5.2 KiB
HTML
POKI_PUT_TOC_HERE
|
|
|
|
<p/>
|
|
<button style="font-weight:bold;color:maroon;border:0" onclick="bodyToggler.expandAll();" href="javascript:;">Expand all sections</button>
|
|
<button style="font-weight:bold;color:maroon;border:0" onclick="bodyToggler.collapseAll();" href="javascript:;">Collapse all sections</button>
|
|
|
|
<p/> As discussed in the section on
|
|
POKI_PUT_LINK_FOR_PAGE(file-formats.html)HERE, Miller supports several
|
|
different file formats. Different tools are good at different things, so
|
|
it’s important to be able to move data into and out of other languages.
|
|
<b>CSV</b> and <b>JSON</b> are well-known, of course; here are some examples
|
|
using <b>DKVP</b> format, with <b>Ruby</b> and <b>Python</b>. Last, we show how
|
|
to use arbitrary <b>shell commands</b> to extend functionality beyond Miller’s
|
|
domain-specific language.
|
|
|
|
<h1>DKVP I/O in Python</h1>
|
|
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_dkvp_python');" href="javascript:;">Toggle section visibility</button>
|
|
<div id="body_section_toggle_dkvp_python" style="display: block">
|
|
|
|
<p/>
|
|
Here are the I/O routines:
|
|
|
|
POKI_INCLUDE_ESCAPED(polyglot-dkvp-io/dkvp_io.py)HERE
|
|
|
|
And here is an example using them:
|
|
|
|
POKI_RUN_COMMAND{{cat polyglot-dkvp-io/example.py}}HERE
|
|
|
|
Run as-is:
|
|
|
|
POKI_RUN_COMMAND{{python polyglot-dkvp-io/example.py < data/small}}HERE
|
|
|
|
Run as-is, then pipe to Miller for pretty-printing:
|
|
|
|
POKI_RUN_COMMAND{{python polyglot-dkvp-io/example.py < data/small | mlr --opprint cat}}HERE
|
|
|
|
</div>
|
|
<h1>DKVP I/O in Ruby</h1>
|
|
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_dkvp_ruby');" href="javascript:;">Toggle section visibility</button>
|
|
<div id="body_section_toggle_dkvp_ruby" style="display: block">
|
|
|
|
<p/>
|
|
Here are the I/O routines:
|
|
|
|
POKI_INCLUDE_ESCAPED(polyglot-dkvp-io/dkvp_io.rb)HERE
|
|
|
|
And here is an example using them:
|
|
|
|
POKI_RUN_COMMAND{{cat polyglot-dkvp-io/example.rb}}HERE
|
|
|
|
Run as-is:
|
|
|
|
POKI_RUN_COMMAND{{ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small}}HERE
|
|
|
|
Run as-is, then pipe to Miller for pretty-printing:
|
|
|
|
POKI_RUN_COMMAND{{ruby -I./polyglot-dkvp-io polyglot-dkvp-io/example.rb data/small | mlr --opprint cat}}HERE
|
|
|
|
</div>
|
|
|
|
<h1>SQL-output examples</h1>
|
|
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_sql_output_examples');" href="javascript:;">Toggle section visibility</button>
|
|
<div id="body_section_toggle_sql_output_examples" style="display: block">
|
|
|
|
<p/>Please see <a href="10-min.html#SQL-output_examples">here</a>.
|
|
|
|
</div>
|
|
|
|
<h1>SQL-input examples</h1>
|
|
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_sql_input_examples');" href="javascript:;">Toggle section visibility</button>
|
|
<div id="body_section_toggle_sql_input_examples" style="display: block">
|
|
|
|
<p/>Please see <a href="10-min.html#SQL-input_examples">here</a>.
|
|
|
|
</div>
|
|
<h1>Running shell commands</h1>
|
|
<button style="font-weight:bold;color:maroon;border:0" padding=0 onclick="bodyToggler.toggle('body_section_toggle_shell_commands');" href="javascript:;">Toggle section visibility</button>
|
|
<div id="body_section_toggle_shell_commands" style="display: block">
|
|
|
|
<p/>The <a href="reference-dsl.html#system">system</a> DSL function allows you to run a specific shell command and put its output — minus the final newline — into a record field. The command itself is any string, either a literal string, or a concatenation of strings, perhaps including other field values or what have you.
|
|
|
|
POKI_RUN_COMMAND{{mlr --opprint put '$o = system("echo hello world")' data/small}}HERE
|
|
POKI_RUN_COMMAND{{mlr --opprint put '$o = system("echo {" . NR . "}")' data/small}}HERE
|
|
POKI_RUN_COMMAND{{mlr --opprint put '$o = system("echo -n ".$a."| sha1sum")' data/small}}HERE
|
|
|
|
<p/>Note that running a subprocess on every record takes a non-trivial amount of time. Comparing asking the system <code>date</code> command for the current time in nanoseconds versus computing it in process:
|
|
|
|
<!-- hard-coded since %N doesn't exist on all platforms -->
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint put '$t=system("date +%s.%N")' then step -a delta -f t data/small
|
|
a b i x y t t_delta
|
|
pan pan 1 0.3467901443380824 0.7268028627434533 1568774318.513903817 0
|
|
eks pan 2 0.7586799647899636 0.5221511083334797 1568774318.514722876 0.000819
|
|
wye wye 3 0.20460330576630303 0.33831852551664776 1568774318.515618046 0.000895
|
|
eks wye 4 0.38139939387114097 0.13418874328430463 1568774318.516547441 0.000929
|
|
wye pan 5 0.5732889198020006 0.8636244699032729 1568774318.517518828 0.000971
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
<p/>
|
|
<div class="pokipanel">
|
|
<pre>
|
|
$ mlr --opprint put '$t=systime()' then step -a delta -f t data/small
|
|
a b i x y t t_delta
|
|
pan pan 1 0.3467901443380824 0.7268028627434533 1568774318.518699 0
|
|
eks pan 2 0.7586799647899636 0.5221511083334797 1568774318.518717 0.000018
|
|
wye wye 3 0.20460330576630303 0.33831852551664776 1568774318.518723 0.000006
|
|
eks wye 4 0.38139939387114097 0.13418874328430463 1568774318.518727 0.000004
|
|
wye pan 5 0.5732889198020006 0.8636244699032729 1568774318.518730 0.000003
|
|
</pre>
|
|
</div>
|
|
<p/>
|
|
|
|
</div>
|