diff --git a/docs/10min.rst b/docs/10min.rst index 4fe1ffad1..2f56168b8 100644 --- a/docs/10min.rst +++ b/docs/10min.rst @@ -284,14 +284,16 @@ Choices for printing to files Often we want to print output to the screen. Miller does this by default, as we've seen in the previous examples. -Sometimes we want to print output to another file: just use **> outputfilenamegoeshere** at the end of your command:: +Sometimes we want to print output to another file: just use **> outputfilenamegoeshere** at the end of your command: + +:: % mlr --icsv --opprint cat example.csv > newfile.csv # Output goes to the new file; # nothing is printed to the screen. - - - 
+
+::
+
     % cat newfile.csv
     color  shape    flag index quantity rate
     yellow triangle 1    11    43.6498  9.8870
@@ -382,7 +384,6 @@ For example, if you have::
     shape,flag,index
     circle,1,24
     square,0,36
-
then that's a way of saying:: diff --git a/docs/10min.rst.in b/docs/10min.rst.in index 282b40b4b..13dfca49e 100644 --- a/docs/10min.rst.in +++ b/docs/10min.rst.in @@ -100,14 +100,16 @@ Choices for printing to files Often we want to print output to the screen. Miller does this by default, as we've seen in the previous examples. -Sometimes we want to print output to another file: just use **> outputfilenamegoeshere** at the end of your command:: +Sometimes we want to print output to another file: just use **> outputfilenamegoeshere** at the end of your command: + +:: % mlr --icsv --opprint cat example.csv > newfile.csv # Output goes to the new file; # nothing is printed to the screen. -
- - 
+
+::
+
     % cat newfile.csv
     color  shape    flag index quantity rate
     yellow triangle 1    11    43.6498  9.8870
@@ -185,7 +187,6 @@ For example, if you have::
     shape,flag,index
     circle,1,24
     square,0,36
-
then that's a way of saying:: diff --git a/docs/README.md b/docs/README.md index 1c6912a32..14816a12c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -38,5 +38,5 @@ ## To do * Let's all discuss if/how we want the v2 docs to be structured better than the v1 docs. -* Scan for non-ported markup * Autogen the `manpage.txt` +* `make linkcheck` diff --git a/docs/cookbook2.rst b/docs/cookbook2.rst index aba6f59bf..8f3266da3 100644 --- a/docs/cookbook2.rst +++ b/docs/cookbook2.rst @@ -22,7 +22,6 @@ Given this `word list `_, first take a look to see what aba abac abaca -
Then the following will randomly sample ten words with four to eight characters in them: @@ -523,6 +522,5 @@ But using a very small font size (as small as my Mac will let me go), and by cho iwidth=$(stty size | mlr --nidx --fs space cut -f 2) echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth" \ | mlr put -f programs/mand.mlr - .. image:: pix/mand.png diff --git a/docs/cookbook2.rst.in b/docs/cookbook2.rst.in index 1fb4f525b..f40445f18 100644 --- a/docs/cookbook2.rst.in +++ b/docs/cookbook2.rst.in @@ -19,7 +19,6 @@ Given this `word list `_, first take a look to see what aba abac abaca - Then the following will randomly sample ten words with four to eight characters in them: @@ -161,6 +160,5 @@ But using a very small font size (as small as my Mac will let me go), and by cho iwidth=$(stty size | mlr --nidx --fs space cut -f 2) echo "rcorn=-1.755350,icorn=+0.014230,side=0.000020,maxits=10000,iheight=$iheight,iwidth=$iwidth" \ | mlr put -f programs/mand.mlr - .. image:: pix/mand.png diff --git a/docs/reference-dsl.rst b/docs/reference-dsl.rst index bdab8af69..698b2d2bb 100644 --- a/docs/reference-dsl.rst +++ b/docs/reference-dsl.rst @@ -859,10 +859,6 @@ Type-test and type-assertion expressions The following ``is...`` functions take a value and return a boolean indicating whether the argument is of the indicated type. The ``assert_...`` functions return their argument if it is of the specified type, and cause a fatal error otherwise: - - - - - -
- :: $ mlr -F | grep ^is @@ -883,9 +879,6 @@ The following ``is...`` functions take a value and return a boolean indicating w is_present is_string - - :: $ mlr -F | grep ^assert @@ -906,10 +899,6 @@ The following ``is...`` functions take a value and return a boolean indicating w asserting_present asserting_string -
- Please see :ref:`cookbook-data-cleaning-examples` for examples of how to use these. Type-declarations for local variables, function parameter, and function return values @@ -1503,7 +1492,7 @@ Miller's ``while`` and ``do-while`` are unsurprising in comparison to various la x=1,y=2,3=,4=,5=,foo=bar A ``break`` or ``continue`` within nested conditional blocks or if-statements will, of course, propagate to the innermost loop enclosing them, if any. A ``break`` or ``continue`` outside a loop is a syntax error that will be flagged as soon as the expression is parsed, before any input records are ingested. -The existence of ``while``, ``do-while``, and ``for`` loops in Miller's DSL means that you can create infinite-loop scenarios inadvertently. In particular, please recall that DSL statements are executed once if in ``begin`` or ``end`` blocks, and once *per record* otherwise. For example, **``while (NR < 10)`` will never terminate as ``NR`` is only incremented between records**. +The existence of ``while``, ``do-while``, and ``for`` loops in Miller's DSL means that you can create infinite-loop scenarios inadvertently. In particular, please recall that DSL statements are executed once if in ``begin`` or ``end`` blocks, and once *per record* otherwise. For example, **while (NR < 10) will never terminate as NR is only incremented between records**. For-loops ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -1705,8 +1694,6 @@ That's confusing in the abstract, so a concrete example is in order. Suppose the Then we can get at various values as follows: -
- :: $ mlr -n put --jknquoteint -q ' @@ -1729,8 +1716,6 @@ Then we can get at various values as follows: key=3,valuetype=map key=6,valuetype=map - - :: $ mlr -n put --jknquoteint -q ' @@ -1753,8 +1738,6 @@ Then we can get at various values as follows: key1=3,key2=4,valuetype=int key1=6,key2=7,valuetype=map - - :: $ mlr -n put --jknquoteint -q ' @@ -1776,8 +1759,6 @@ Then we can get at various values as follows: ' key1=7,key2=8,valuetype=int -
- C-style triple-for loops ................................................................ @@ -4917,7 +4898,7 @@ The ``syntax error`` message is cryptic: it says ``syntax error at `` followed b Now for transparency: -* As in any language, you can do (see :ref:`reference-dsl-print-statements`) ``print`` (or ``eprint`` to print to stderr). See also :ref:`reference-dsl-dump-statements` and :ref:`reference-dsl-emit-statements`. +* As in any language, you can do (see :ref:`reference-dsl-print-statements`) ``print`` (or ``eprint`` to print to stderr). See also :ref:`reference-dsl-dump-statements` and :ref:`reference-dsl-emit-statements`. * The ``-v`` option to ``mlr put`` and ``mlr filter`` prints abstract syntax trees for your code. While not all details here will be of interest to everyone, certainly this makes questions such as operator precedence completely unambiguous. diff --git a/docs/reference-dsl.rst.in b/docs/reference-dsl.rst.in index e698c9c56..127d3be06 100644 --- a/docs/reference-dsl.rst.in +++ b/docs/reference-dsl.rst.in @@ -443,25 +443,14 @@ Type-test and type-assertion expressions The following ``is...`` functions take a value and return a boolean indicating whether the argument is of the indicated type. The ``assert_...`` functions return their argument if it is of the specified type, and cause a fatal error otherwise: - - - - - -
- :: POKI_RUN_COMMAND{{mlr -F | grep ^is}}HERE - - :: POKI_RUN_COMMAND{{mlr -F | grep ^assert}}HERE -
- Please see :ref:`cookbook-data-cleaning-examples` for examples of how to use these. Type-declarations for local variables, function parameter, and function return values @@ -685,7 +674,7 @@ POKI_INCLUDE_AND_RUN_ESCAPED(data/while-example-1.sh)HERE POKI_INCLUDE_AND_RUN_ESCAPED(data/while-example-2.sh)HERE A ``break`` or ``continue`` within nested conditional blocks or if-statements will, of course, propagate to the innermost loop enclosing them, if any. A ``break`` or ``continue`` outside a loop is a syntax error that will be flagged as soon as the expression is parsed, before any input records are ingested. -The existence of ``while``, ``do-while``, and ``for`` loops in Miller's DSL means that you can create infinite-loop scenarios inadvertently. In particular, please recall that DSL statements are executed once if in ``begin`` or ``end`` blocks, and once *per record* otherwise. For example, **``while (NR < 10)`` will never terminate as ``NR`` is only incremented between records**. +The existence of ``while``, ``do-while``, and ``for`` loops in Miller's DSL means that you can create infinite-loop scenarios inadvertently. In particular, please recall that DSL statements are executed once if in ``begin`` or ``end`` blocks, and once *per record* otherwise. For example, **while (NR < 10) will never terminate as NR is only incremented between records**. For-loops ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -754,26 +743,18 @@ POKI_INCLUDE_AND_RUN_ESCAPED(data/for-oosvar-example-0b.sh)HERE Then we can get at various values as follows: -
- :: POKI_INCLUDE_AND_RUN_ESCAPED(data/for-oosvar-example-0c.sh)HERE - - :: POKI_INCLUDE_AND_RUN_ESCAPED(data/for-oosvar-example-0d.sh)HERE - - :: POKI_INCLUDE_AND_RUN_ESCAPED(data/for-oosvar-example-0e.sh)HERE -
- C-style triple-for loops ................................................................ @@ -1200,7 +1181,7 @@ The ``syntax error`` message is cryptic: it says ``syntax error at `` followed b Now for transparency: -* As in any language, you can do (see :ref:`reference-dsl-print-statements`) ``print`` (or ``eprint`` to print to stderr). See also :ref:`reference-dsl-dump-statements` and :ref:`reference-dsl-emit-statements`. +* As in any language, you can do (see :ref:`reference-dsl-print-statements`) ``print`` (or ``eprint`` to print to stderr). See also :ref:`reference-dsl-dump-statements` and :ref:`reference-dsl-emit-statements`. * The ``-v`` option to ``mlr put`` and ``mlr filter`` prints abstract syntax trees for your code. While not all details here will be of interest to everyone, certainly this makes questions such as operator precedence completely unambiguous. diff --git a/docs/reference-verbs.rst b/docs/reference-verbs.rst index 2d006ca73..a9a8d3d8c 100644 --- a/docs/reference-verbs.rst +++ b/docs/reference-verbs.rst @@ -190,7 +190,6 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua purple 0.496523 1192 green 0.507569 1111 orange 0.468014 292 - .. _reference-verbs-cat: diff --git a/docs/reference-verbs.rst.in b/docs/reference-verbs.rst.in index 2a2d7260c..dddf5a147 100644 --- a/docs/reference-verbs.rst.in +++ b/docs/reference-verbs.rst.in @@ -134,7 +134,6 @@ The canonical use for bootstrap sampling is to put error bars on statistical qua purple 0.496523 1192 green 0.507569 1111 orange 0.468014 292 - .. _reference-verbs-cat: diff --git a/docs/reference.rst b/docs/reference.rst index ae2cd874d..6952c6700 100644 --- a/docs/reference.rst +++ b/docs/reference.rst @@ -430,7 +430,7 @@ with the exception that the ``min`` and ``max`` functions are special: if one ar $ echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)' x=,y=3,a=3,b=3 -* Functions of *absent* variables (e.g. ``mlr put '$y = log10($nonesuch)'``) evaluate to absent, and arithmetic/bitwise/boolean operators with both operands being absent evaluate to absent. Arithmetic operators with one absent operand return the other operand. More specifically, absent values act like zero for addition/subtraction, and one for multiplication: Furthermore, **any expression which evaluates to absent is not stored in the left-hand side of an assignment statement **: +* Functions of *absent* variables (e.g. ``mlr put '$y = log10($nonesuch)'``) evaluate to absent, and arithmetic/bitwise/boolean operators with both operands being absent evaluate to absent. Arithmetic operators with one absent operand return the other operand. More specifically, absent values act like zero for addition/subtraction, and one for multiplication: Furthermore, **any expression which evaluates to absent is not stored in the left-hand side of an assignment statement**: :: @@ -519,7 +519,7 @@ See also https://en.wikipedia.org/wiki/Escape_sequences_in_C. These replacements apply only to strings you key in for the DSL expressions for ``filter`` and ``put``: that is, if you type ``\t`` in a string literal for a ``filter``/``put`` expression, it will be turned into a tab character. If you want a backslash followed by a ``t``, then please type ``\\t``. -However, these replacements are not done automatically within your data stream. If you wish to make these replacements, you can do, for example, for a field named ``field``, `` mlr put '$field = gsub($field, "\\t", "\t")'``. If you need to make such a replacement for all fields in your data, you should probably simply use the system ``sed`` command. +However, these replacements are not done automatically within your data stream. If you wish to make these replacements, you can do, for example, for a field named ``field``, ``mlr put '$field = gsub($field, "\\t", "\t")'``. If you need to make such a replacement for all fields in your data, you should probably simply use the system ``sed`` command. Regular expressions ---------------------------------------------------------------- @@ -618,7 +618,7 @@ The sum, difference, and product of integers is again integer, except for when t The short of it is that Miller does this transparently for you so you needn't think about it. -Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -263 or greater than 263-1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range: +Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\ :sup:`63` or greater than 2\ :sup:`63`\ -1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range: :: @@ -633,7 +633,7 @@ Implementation details of this, for the interested: integer adds and subtracts o 0x7ffffffffffffffe 9223372036854775806 9223372036854775808.000000 0x8000000000000000 0x7fffffffffffffff 9223372036854775807 9223372036854775808.000000 0x8000000000000000 -That is, one cannot check an integer product to see if it is precisely greater than 263-1 or less than -263 using either integer arithmetic (it may have already overflowed) or using double-precision (due to granularity). Instead Miller checks for overflow in 64-bit integer multiplication by seeing whether the absolute value of the double-precision product exceeds the largest representable IEEE double less than 263, which we see from the listing above is 9223372036854774784. (An alternative would be to do all integer multiplies using handcrafted multi-word 128-bit arithmetic. This approach is not taken.) +That is, one cannot check an integer product to see if it is precisely greater than 2\ :sup:`63`\ -1 or less than -2\ :sup:`63` using either integer arithmetic (it may have already overflowed) or using double-precision (due to granularity). Instead Miller checks for overflow in 64-bit integer multiplication by seeing whether the absolute value of the double-precision product exceeds the largest representable IEEE double less than 2\ :sup:`63`, which we see from the listing above is 9223372036854774784. (An alternative would be to do all integer multiplies using handcrafted multi-word 128-bit arithmetic. This approach is not taken.) Pythonic division ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/reference.rst.in b/docs/reference.rst.in index d7c338e5e..3949a11af 100644 --- a/docs/reference.rst.in +++ b/docs/reference.rst.in @@ -318,7 +318,7 @@ with the exception that the ``min`` and ``max`` functions are special: if one ar POKI_RUN_COMMAND{{echo 'x=,y=3' | mlr put '$a=min($x,$y);$b=max($x,$y)'}}HERE -* Functions of *absent* variables (e.g. ``mlr put '$y = log10($nonesuch)'``) evaluate to absent, and arithmetic/bitwise/boolean operators with both operands being absent evaluate to absent. Arithmetic operators with one absent operand return the other operand. More specifically, absent values act like zero for addition/subtraction, and one for multiplication: Furthermore, **any expression which evaluates to absent is not stored in the left-hand side of an assignment statement **: +* Functions of *absent* variables (e.g. ``mlr put '$y = log10($nonesuch)'``) evaluate to absent, and arithmetic/bitwise/boolean operators with both operands being absent evaluate to absent. Arithmetic operators with one absent operand return the other operand. More specifically, absent values act like zero for addition/subtraction, and one for multiplication: Furthermore, **any expression which evaluates to absent is not stored in the left-hand side of an assignment statement**: :: @@ -381,7 +381,7 @@ See also https://en.wikipedia.org/wiki/Escape_sequences_in_C. These replacements apply only to strings you key in for the DSL expressions for ``filter`` and ``put``: that is, if you type ``\t`` in a string literal for a ``filter``/``put`` expression, it will be turned into a tab character. If you want a backslash followed by a ``t``, then please type ``\\t``. -However, these replacements are not done automatically within your data stream. If you wish to make these replacements, you can do, for example, for a field named ``field``, `` mlr put '$field = gsub($field, "\\t", "\t")'``. If you need to make such a replacement for all fields in your data, you should probably simply use the system ``sed`` command. +However, these replacements are not done automatically within your data stream. If you wish to make these replacements, you can do, for example, for a field named ``field``, ``mlr put '$field = gsub($field, "\\t", "\t")'``. If you need to make such a replacement for all fields in your data, you should probably simply use the system ``sed`` command. Regular expressions ---------------------------------------------------------------- @@ -475,7 +475,7 @@ The sum, difference, and product of integers is again integer, except for when t The short of it is that Miller does this transparently for you so you needn't think about it. -Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -263 or greater than 263-1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range: +Implementation details of this, for the interested: integer adds and subtracts overflow by at most one bit so it suffices to check sign-changes. Thus, Miller allows you to add and subtract arbitrary 64-bit signed integers, converting only to float precisely when the result is less than -2\ :sup:`63` or greater than 2\ :sup:`63`\ -1. Multiplies, on the other hand, can overflow by a word size and a sign-change technique does not suffice to detect overflow. Instead Miller tests whether the floating-point product exceeds the representable integer range. Now, 64-bit integers have 64-bit precision while IEEE-doubles have only 52-bit mantissas -- so, there are 53 bits including implicit leading one. The following experiment explicitly demonstrates the resolution at this range: :: @@ -490,7 +490,7 @@ Implementation details of this, for the interested: integer adds and subtracts o 0x7ffffffffffffffe 9223372036854775806 9223372036854775808.000000 0x8000000000000000 0x7fffffffffffffff 9223372036854775807 9223372036854775808.000000 0x8000000000000000 -That is, one cannot check an integer product to see if it is precisely greater than 263-1 or less than -263 using either integer arithmetic (it may have already overflowed) or using double-precision (due to granularity). Instead Miller checks for overflow in 64-bit integer multiplication by seeing whether the absolute value of the double-precision product exceeds the largest representable IEEE double less than 263, which we see from the listing above is 9223372036854774784. (An alternative would be to do all integer multiplies using handcrafted multi-word 128-bit arithmetic. This approach is not taken.) +That is, one cannot check an integer product to see if it is precisely greater than 2\ :sup:`63`\ -1 or less than -2\ :sup:`63` using either integer arithmetic (it may have already overflowed) or using double-precision (due to granularity). Instead Miller checks for overflow in 64-bit integer multiplication by seeing whether the absolute value of the double-precision product exceeds the largest representable IEEE double less than 2\ :sup:`63`, which we see from the listing above is 9223372036854774784. (An alternative would be to do all integer multiplies using handcrafted multi-word 128-bit arithmetic. This approach is not taken.) Pythonic division ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/go/todo.txt b/go/todo.txt index 329d30ddd..d5551f0ae 100644 --- a/go/todo.txt +++ b/go/todo.txt @@ -17,6 +17,8 @@ TOP OF LIST: * clean up 'mlr --help' to not advertise things it doesn't do yet +* prohibit NR etc at LHS within the CST + * ASV? golang csv package parameterizes comma but not crlf :( * mlr -f / -F