# Arrays

Miller data types are listed on the [Data types](reference-main-data-types.md)
page; here we focus specifically on arrays.

Arrays are supported [as of Miller 6](new-in-miller-6.md), and constitute one
of the major advantages of Miller 6.

## Array literals

Array literals are written in square brackets braces with integer indices. Array slots can be any [Miller data type](reference-main-data-types.md) (including other arrays, or maps).

GENMD-RUN-COMMAND
mlr -n put '
  end {
    x = [ "a", 1, "b", {"x": 2, "y": [3,4,5]}, 99, true];
    print x;
  }
'
GENMD-EOF

As with maps and argument-lists, trailing commas are supported:

GENMD-RUN-COMMAND
mlr -n put '
  end {
    x = [
      "a",
      "b",
      "c",
    ];
    print x;
  }
'
GENMD-EOF

Also note that several [built-in functions](reference-dsl-builtin-functions.md) operate on arrays and/or return arrays.

## 1-up indexing

The most important difference between Miller's arrays and arrays in other
languages is that indices start with 1, not 0.  (The same is true for [Miller
strings](reference-main-strings.md).) This is intentional.

1-up array indices may feel like a thing of the past, belonging to Fortran and
Matlab, say; or R and Julia as well, which are more modern.  But the overall
trend is decidedly toward 0-up. This means that if Miller does 1-up array
indices, it should do so for good reasons.

When arrays were introduced into [Miller 6](new-in-miller-6.md), it quickly became
clear that 1-up indexing is the right thing for Miller.  So many other things
are already 1-up in Miller, and always have been, mostly inherited from AWK:

* The `awk`-like [built-in variables](reference-dsl-variables.md#built-in-variables) `NF`, `NR`, and `FNR` are 1-up in Miller. So for idioms like `@records[NR] = $*` it's natural to index from 1; `@records[NR-1] = $*` would be error-prone and would result in frequent off-by-one errors.
* In particular, fields have always been indexed 1-up for [NIDX and DKVP formats](file-formats.md).
* [Regex captures](reference-main-regular-expressions.md) run from `"\1"` to `"\9"` (`"\0"` is the entire match substring).

## Negative-index aliasing

Imitating Python and other languages, you can use negative indices to read backward from the end of the array,
while positive indices read forward from the start. If an array has length `n` then `-n..-1` are aliases for `1..n`, respectively; 0 is never a valid array index in Miller.

GENMD-RUN-COMMAND
mlr -n put '
  end {
    x = [10, 20, 30, 40, 50];
    print x[1];
    print x[-1];
    print x[1:2];
    print x[-2:-1];
  }
'
GENMD-EOF

## Slicing

Miller supports slicing using `[lo:hi]` syntax.  Either or both of the indices
in a slice can be negatively aliased as described above.  Unlike in Python,
Miller array-slice indices are inclusive on both sides: `x[3:5]` means `[x[3],
x[4], x[5]]`.

GENMD-RUN-COMMAND
mlr -n put '
  end {
    x = [10, 20, 30, 40, 50];
    print x[3:4];
    print x[:2];
    print x[3:];
    print x[1:-1];
    print x[2:-2];
  }
'
GENMD-EOF

## Out-of-bounds indexing

Out-of-bounds index accesses are [absent](reference-main-null-data.md), but out-of-bounds slice
accesses result in trimming the indices, resulting in a short array or even the empty array.  (This
behavior intentionally imitates Python.)

GENMD-RUN-COMMAND
mlr -n put '
  end {
    x = [10, 20, 30, 40, 50];
    print x[1];
    print x[5];
    print x[6]; # absent
  }
'
GENMD-EOF

GENMD-RUN-COMMAND
mlr -n put '
  end {
    x = [10, 20, 30, 40, 50];
    print x[1:2];
    print x[1:6];
    print x[10:20];
  }
'
GENMD-EOF

## Auto-create results in maps

As noted on the [maps page](reference-main-maps.md), indexing any
as-yet-assigned local variable or out-of-stream variable results in
**auto-create** of that variable as a map variable:

GENMD-RUN-COMMAND
mlr --csv --from example.csv put -q '
  # You can do this but you do not need to:
  # begin { @last_rates = {} }
  @last_rates[$shape] = $rate;
  end {
    dump @last_rates;
  }
'
GENMD-EOF

*This also means that auto-create results in maps, not arrays, even if keys are integers.*
If you want to auto-extend an [array](reference-main-arrays.md), initialize it explicitly to `[]`.

GENMD-RUN-COMMAND
mlr --csv --from example.csv head -n 4 then put -q '
  begin {
    @my_array = [];
  }
  @my_array[NR] = $quantity;
  @my_map[NR] = $rate;
  end {
    dump
  }
'
GENMD-EOF

## Auto-extend and null-gaps

Once an array is initialized, it can be extended by assigning to indices beyond
its length.  If each write is one past the end of the array, the array will
grow by one. (Memory management, handled for you, is careful handled here in
Miller: not to worry, capacity is doubled so performance doesn't suffer a
rellocate on every single extend.)

This is important in Miller so you can do things like `@records[NR] = $*` with
a minimum of keystrokes without worrying about explicitly resizing arrays. In
particular, you can iteratively populate arrays as you read your data files,
without having to first know how many records they have.

However, if an array is written to more than one past its end, [values of type
JSON-null](reference-main-data-types.md) are used to fill in the gaps. These
are called **null-gaps**.

GENMD-RUN-COMMAND
mlr -n put '
  end {
    no_gaps = [];
    no_gaps[1] = "a";
    no_gaps[2] = "b";

    gaps = [];
    gaps[1] = "a";
    gaps[5] = "e";

    print no_gaps;
    print gaps;
  }
'
GENMD-EOF

## Unset as shift

Unsetting an array index results in shifting all higher-index elements down by one:

GENMD-RUN-COMMAND
mlr -n put '
  end {
    x = [ "a", "b", "c", "d", "e"];
    print x;
    unset x[2];
    print x;
  }
'
GENMD-EOF

More generally, you can get shift and pop operations by unsetting indices 1 and -1:

GENMD-CARDIFY-HIGHLIGHT-ONE
$ mlr repl -q
[mlr] x=[1,2,3,4,5]
[mlr] unset x[-1]
[mlr] x
[1, 2, 3, 4]
[mlr] unset x[-1]
[mlr] x
[1, 2, 3]
[mlr]
[mlr] x=[1,2,3,4,5]
[mlr] unset x[1]
[mlr] x
[2, 3, 4, 5]
[mlr] unset x[1]
[mlr] x
[3, 4, 5]
[mlr]
GENMD-EOF

## Looping

See [single-variable for-loops](reference-dsl-control-structures.md#single-variable-for-loops) and [key-value for-loops](reference-dsl-control-structures.md#key-value-for-loops).

## Array-valued fields in CSV files

See the [flatten/unflatten page](flatten-unflatten.md).
