# DSL higher-order functions

A _higher-order function_ is one which takes [another function](reference-dsl-user-defined-functions.md)
as an argument.
As of [Miller 6](new-in-miller-6.md) you can use
[`select`](reference-dsl-builtin-functions.md#select),
[`apply`](reference-dsl-builtin-functions.md#apply),
[`reduce`](reference-dsl-builtin-functions.md#reduce),
[`fold`](reference-dsl-builtin-functions.md#fold), and
[`sort`](reference-dsl-builtin-functions.md#sort), and
[`any`](reference-dsl-builtin-functions.md#any), and
[`every`](reference-dsl-builtin-functions.md#every) to express flexible,
intuitive operations on arrays and maps, as an alternative to things which
would otherwise require for-loops.

See also the [`get_keys`](reference-dsl-builtin-functions.md#get_keys) and [`get_values`](reference-dsl-builtin-functions.md#get_values) functions which, when given a map, return an array of its keys or an array of its values, respectively.

## select

The [`select`](reference-dsl-builtin-functions.md#select) function takes a map or array as its first argument and a function as its second argument.  It includes each input element in the output if the function returns true.

For arrays, that function should take one argument, for an array element; for maps, it should take two, for a map element key and value. In either case, it should return a boolean.

A perhaps helpful analogy: the `select` function is to arrays and maps as the [`filter`](reference-verbs.md#filter) is to records.

Array examples:

GENMD-RUN-COMMAND
mlr -n put '
  end {
    my_array = [2, 9, 10, 3, 1, 4, 5, 8, 7, 6];

    print "Original:";
    print my_array;

    print;
    print "Evens:";
    print select(my_array, func (e) { return e % 2 == 0});

    print;
    print "Odds:";
    print select(my_array, func (e) { return e % 2 == 1});
    print;
  }
'
GENMD-EOF

Map examples:

GENMD-RUN-COMMAND
mlr -n put '
  end {
    my_map = {"cubit": 823, "dale": 13, "apple": 199, "ember": 191, "bottle": 107};
    print "Original:";
    print my_map;

    print;
    print "Keys with an 'o' in them:";
    print select(my_map, func (k,v) { return k =~ "o"});

    print;
    print "Values with last digit >= 5:";
    print select(my_map, func (k,v) { return v % 10 >= 5});
  }
'
GENMD-EOF

## apply

The [`apply`](reference-dsl-builtin-functions.md#apply) function takes a map or array as its first argument and a function as its second argument.  It applies the function to each element of the array or map.

For arrays, the function should take one argument, representing an array element, and return a new element. For maps, it should take two, for the map element key and value. It should return a new key-value pair (i.e., a single-entry map).

A perhaps helpful analogy: the `apply` function is to arrays and maps as the [`put`](reference-verbs.md#put) is to records.

Array examples:

GENMD-RUN-COMMAND
mlr -n put '
  end {
    my_array = [2, 9, 10, 3, 1, 4, 5, 8, 7, 6];
    print "Original:";
    print my_array;

    print;
    print "Squares:";
    print apply(my_array, func(e) { return e**2 });

    print;
    print "Cubes:";
    print apply(my_array, func(e) { return e**3 });

    print;
    print "Sorted cubes:";
    print sort(apply(my_array, func(e) { return e**3 }));
  }
'
GENMD-EOF

GENMD-RUN-COMMAND
mlr -n put '
  end {
    my_map = {"cubit": 823, "dale": 13, "apple": 199, "ember": 191, "bottle": 107};
    print "Original:";
    print my_map;

    print;
    print "Squared values:";
    print apply(my_map, func(k,v) { return {k: v**2} });

    print;
    print "Cubed values, sorted by key:";
    print sort(apply(my_map, func(k,v) { return {k: v**3} }));

    print;
    print "Same, with upcased keys:";
    print sort(apply(my_map, func(k,v) { return {toupper(k): v**3} }));
  }
'
GENMD-EOF

## reduce

The [`reduce`](reference-dsl-builtin-functions.md#reduce) function takes a map or array as its first argument and a function as its second argument. It accumulates entries into a final output, such as a sum or product.

For arrays, the function should take two arguments, for the accumulated value and the array element; for maps, it should take four, for the accumulated key and value, and the map-element key and value. In either case it should return the updated accumulator.

The start value for the accumulator is the first element for arrays, or the first element's key-value pair for maps.

GENMD-RUN-COMMAND
mlr -n put '
  end {
    my_array = [2, 9, 10, 3, 1, 4, 5, 8, 7, 6];

    print "Original:";
    print my_array;

    print;
    print "First element:";
    print reduce(my_array, func (acc,e) { return acc });

    print;
    print "Last element:";
    print reduce(my_array, func (acc,e) { return e });

    print;
    print "Sum of values:";
    print reduce(my_array, func (acc,e) { return acc + e });

    print;
    print "Product of values:";
    print reduce(my_array, func (acc,e) { return acc * e });

    print;
    print "Concatenation of values:";
    print reduce(my_array, func (acc,e) { return acc. "," . e });
  }
'
GENMD-EOF

GENMD-RUN-COMMAND
mlr -n put '
  end {
    my_map = {"cubit": 823, "dale": 13, "apple": 199, "ember": 191, "bottle": 107};
    print "Original:";
    print my_map;

    print;
    print "First key-value pair:";
    print reduce(my_map, func (acck,accv,ek,ev) { return {acck: accv}});

    print;
    print "Last key-value pair:";
    print reduce(my_map, func (acck,accv,ek,ev) { return {ek: ev}});

    print;
    print "Concatenate keys and values:";
    print reduce(my_map, func (acck,accv,ek,ev) { return {acck . "," . ek: accv . "," . ev}});

    print;
    print "Sum of values:";
    print reduce(my_map, func (acck,accv,ek,ev) { return {"sum": accv + ev }});

    print;
    print "Product of values:";
    print reduce(my_map, func (acck,accv,ek,ev) { return {"product": accv * ev }});

    print;
    print "String-join of values:";
    print reduce(my_map, func (acck,accv,ek,ev) { return {"joined": accv . "," . ev }});
  }
'
GENMD-EOF

## fold

The [`fold`](reference-dsl-builtin-functions.md#fold) function is the same as `reduce`, except that instead of the starting value for the accumulation being taken from the first entry of the array/map, you specify it as the third argument.

GENMD-RUN-COMMAND
mlr -n put '
  end {
    my_array = [2, 9, 10, 3, 1, 4, 5, 8, 7, 6];

    print "Original:";
    print my_array;

    print;
    print "Sum with reduce:";
    print reduce(my_array, func (acc,e) { return acc + e });

    print;
    print "Sum with fold and 0 initial value:";
    print fold(my_array, func (acc,e) { return acc + e }, 0);

    print;
    print "Sum with fold and 1000000 initial value:";
    print fold(my_array, func (acc,e) { return acc + e }, 1000000);
  }
'
GENMD-EOF

GENMD-RUN-COMMAND
mlr -n put '
  end {
    my_map = {"cubit": 823, "dale": 13, "apple": 199, "ember": 191, "bottle": 107};
    print "Original:";
    print my_map;

    print;
    print "First key-value pair -- note this is the starting accumulator:";
    print fold(my_map, func (acck,accv,ek,ev) { return {acck: accv}}, {"start": 999});

    print;
    print "Last key-value pair:";
    print fold(my_map, func (acck,accv,ek,ev) { return {ek: ev}}, {"start": 999});

    print;
    print "Sum of values with fold and 0 initial value:";
    print fold(my_map, func (acck,accv,ek,ev) { return {"sum": accv + ev} }, {"sum": 0});

    print;
    print "Sum of values with fold and 1000000 initial value:";
    print fold(my_map, func (acck,accv,ek,ev) { return {"sum": accv + ev} }, {"sum": 1000000});
  }
'
GENMD-EOF

## sort

The [`sort`](reference-dsl-builtin-functions.md#sort) function takes a map or array as its first argument, and it can take a function as its second argument. Unlike the other higher-order functions, the second argument can be omitted when the natural ordering is desired -- ordered by array element for arrays, or by key for maps.

As a second option, character flags such as `r` for reverse or `c` for case-folded lexical sort can be supplied as the second argument.

As a third option, a function can be supplied as the second argument.

For arrays, that function should take two arguments `a` and `b`, returning a negative, zero, or positive number as `a<b`, `a==b`, or `a>b` respectively. For maps, the function should take four arguments `ak`, `av`, `bk`, and `bv`, again returning negative, zero, or positive, using `a`'s and `b`'s keys and values.

Array examples:

GENMD-RUN-COMMAND
mlr -n put '
  end {
    my_array = [2, 9, 10, 3, 1, 4, 5, 8, 7, 6];

    print "Original:";
    print my_array;

    print;
    print "Ascending:";
    print sort(my_array);
    print sort(my_array, func (a,b) { return a <=> b });

    print;
    print "Descending:";
    print sort(my_array, "r");
    print sort(my_array, func (a,b) { return b <=> a });
  }
'
GENMD-EOF

Map examples:

GENMD-RUN-COMMAND
mlr -n put '
  end {
    my_map = {"cubit": 823, "dale": 13, "apple": 199, "ember": 191, "bottle": 107};

    print "Original:";
    print my_map;

    print;
    print "Ascending by key:";
    print sort(my_map);
    print sort(my_map, func(ak,av,bk,bv) { return ak <=> bk });

    print;
    print "Descending by key:";
    print sort(my_map, "r");
    print sort(my_map, func(ak,av,bk,bv) { return bk <=> ak });

    print;
    print "Ascending by value:";
    print sort(my_map, func(ak,av,bk,bv) { return av <=> bv });

    print;
    print "Descending by value:";
    print sort(my_map, func(ak,av,bk,bv) { return bv <=> av });
  }
'
GENMD-EOF

Please see the [sorting page](sorting.md) for more examples.

## any and every

This is a way to do a logical OR/AND, respectively, of several boolean expressions, without the explicit `||`/`&&` and without a `for`-loop. This is a keystroke-saving convenience.

GENMD-RUN-COMMAND
mlr --c2p cat example.csv
GENMD-EOF

GENMD-RUN-COMMAND
mlr --c2p --from example.csv filter 'any({"color":"red","shape":"square"}, func(k,v) {return $[k] == v})'
GENMD-EOF

GENMD-RUN-COMMAND
mlr --c2p --from example.csv filter 'every({"color":"red","shape":"square"}, func(k,v) {return $[k] == v})'
GENMD-EOF

GENMD-RUN-COMMAND
mlr --c2p --from example.csv put '$is_red_square = every({"color":"red","shape":"square"}, func(k,v) {return $[k] == v})'
GENMD-EOF

GENMD-RUN-COMMAND
mlr --c2p --from example.csv filter 'any([16,51,61,64], func(e) {return $index == e})'
GENMD-EOF

This last example could also be done using a map:

GENMD-RUN-COMMAND
mlr --c2p --from example.csv filter '
  begin {
    @indices = {16:true, 51:true, 61:true, 64:true};
  }
  @indices[$index] == true;
'
GENMD-EOF

## Combined examples

Using a paradigm from the [page on operating on all records](operating-on-all-records.md), we can retain a column from the input data as an array, then apply some higher-order functions to it:

GENMD-RUN-COMMAND
mlr --c2p cat example.csv
GENMD-EOF

GENMD-RUN-COMMAND
mlr --c2p --from example.csv put -q '
  begin {
    @indexes = [] # So auto-extend will make an array, not a map
  }
  @indexes[NR] = $index;
  end {

    print "Original:";
    print @indexes;

    print;
    print "Sorted:";
    print sort(@indexes, "r");

    print;
    print "Sorted, then cubed:";
    print apply(
      sort(@indexes, "r"),
      func(e) { return e**3 },
    );

    print;
    print "Sorted, then cubed, then summed:";
    print reduce(
      apply(
        sort(@indexes, "r"),
        func(e) { return e**3 },
      ),
      func(acc, e) { return acc + e },
    )
  }
'
GENMD-EOF

## Caveats

### Remember return

From other languages, it's easy to write accidentally

GENMD-RUN-COMMAND-TOLERATING-ERROR
mlr -n put 'end { print select([1,2,3,4,5], func (e) { e >= 3 })}'
GENMD-EOF

instead of

GENMD-RUN-COMMAND
mlr -n put 'end { print select([1,2,3,4,5], func (e) { return e >= 3 })}'
GENMD-EOF

### No IIFEs

As of September 2021, immediately invoked function expressions (IIFEs) are not part of the Miller DSL's grammar. For example, this doesn't work yet:

GENMD-RUN-COMMAND-TOLERATING-ERROR
mlr -n put '
  end {
    x = 3;
    y = (func (e) { return e**7 })(x);
    print y;
  }
'
GENMD-EOF

but this does:

GENMD-RUN-COMMAND
mlr -n put '
  end {
    x = 3;
    f = func (e) { return e**7 };
    y = f(x);
    print y;
  }
'
GENMD-EOF

### Built-in functions are currently unsupported as arguments

[Built-in functions](reference-dsl-user-defined-functions.md) are, as of
September 2021, a bit separate from [user-defined
functions](reference-dsl-builtin-functions.md) internally to Miller, and can't
be used directly as arguments to higher-order functions.

For example, this doesn't work yet:

GENMD-RUN-COMMAND-TOLERATING-ERROR
mlr -n put '
  end {
    notches = [0,1,2,3];
    radians = apply(notches, func (e) { return e * M_PI / 8 });
    cosines = apply(radians, cos);
    print cosines;
  }
'
GENMD-EOF

but this does:

GENMD-RUN-COMMAND
mlr -n put '
  end {
    notches = [0,1,2,3];
    radians = apply(notches, func (e) { return e * M_PI / 8 });
    # cosines = apply(radians, cos);
    cosines = apply(radians, func (e) { return cos(e) });
    print cosines;
  }
'
GENMD-EOF
