mirror of
https://github.com/johnkerl/miller.git
synced 2026-07-25 08:53:55 +00:00
233 lines
6.2 KiB
Markdown
233 lines
6.2 KiB
Markdown
# Arrays
|
|
|
|
Miller data types are listed on the [Data types](reference-main-data-types.md)
|
|
page; here we focus specifically on arrays.
|
|
|
|
Arrays are supported [as of Miller 6](new-in-miller-6.md), and constitute one
|
|
of the major advantages of Miller 6.
|
|
|
|
## Array literals
|
|
|
|
Array literals are written in square brackets braces with integer indices. Array slots can be any [Miller data type](reference-main-data-types.md) (including other arrays, or maps).
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr -n put '
|
|
end {
|
|
x = [ "a", 1, "b", {"x": 2, "y": [3,4,5]}, 99, true];
|
|
print x;
|
|
}
|
|
'
|
|
GENMD-EOF
|
|
|
|
As with maps and argument-lists, trailing commas are supported:
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr -n put '
|
|
end {
|
|
x = [
|
|
"a",
|
|
"b",
|
|
"c",
|
|
];
|
|
print x;
|
|
}
|
|
'
|
|
GENMD-EOF
|
|
|
|
Also note that several [built-in functions](reference-dsl-builtin-functions.md) operate on arrays and/or return arrays.
|
|
|
|
## 1-up indexing
|
|
|
|
The most important difference between Miller's arrays and arrays in other
|
|
languages is that indices start with 1, not 0. (The same is true for [Miller
|
|
strings](reference-main-strings.md).) This is intentional.
|
|
|
|
1-up array indices may feel like a thing of the past, belonging to Fortran and
|
|
Matlab, say; or R and Julia as well, which are more modern. But the overall
|
|
trend is decidedly toward 0-up. This means that if Miller does 1-up array
|
|
indices, it should do so for good reasons.
|
|
|
|
When arrays were introduced into [Miller 6](new-in-miller-6.md), it quickly became
|
|
clear that 1-up indexing is the right thing for Miller. So many other things
|
|
are already 1-up in Miller, and always have been, mostly inherited from AWK:
|
|
|
|
* The `awk`-like [built-in variables](reference-dsl-variables.md#built-in-variables) `NF`, `NR`, and `FNR` are 1-up in Miller. So for idioms like `@records[NR] = $*` it's natural to index from 1; `@records[NR-1] = $*` would be error-prone and would result in frequent off-by-one errors.
|
|
* In particular, fields have always been indexed 1-up for [NIDX and DKVP formats](file-formats.md).
|
|
* [Regex captures](reference-main-regular-expressions.md) run from `"\1"` to `"\9"` (`"\0"` is the entire match substring).
|
|
|
|
## Negative-index aliasing
|
|
|
|
Imitating Python and other languages, you can use negative indices to read backward from the end of the array,
|
|
while positive indices read forward from the start. If an array has length `n` then `-n..-1` are aliases for `1..n`, respectively; 0 is never a valid array index in Miller.
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr -n put '
|
|
end {
|
|
x = [10, 20, 30, 40, 50];
|
|
print x[1];
|
|
print x[-1];
|
|
print x[1:2];
|
|
print x[-2:-1];
|
|
}
|
|
'
|
|
GENMD-EOF
|
|
|
|
## Slicing
|
|
|
|
Miller supports slicing using `[lo:hi]` syntax. Either or both of the indices
|
|
in a slice can be negatively aliased as described above. Unlike in Python,
|
|
Miller array-slice indices are inclusive on both sides: `x[3:5]` means `[x[3],
|
|
x[4], x[5]]`.
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr -n put '
|
|
end {
|
|
x = [10, 20, 30, 40, 50];
|
|
print x[3:4];
|
|
print x[:2];
|
|
print x[3:];
|
|
print x[1:-1];
|
|
print x[2:-2];
|
|
}
|
|
'
|
|
GENMD-EOF
|
|
|
|
## Out-of-bounds indexing
|
|
|
|
Out-of-bounds index accesses are [absent](reference-main-null-data.md), but out-of-bounds slice
|
|
accesses result in trimming the indices, resulting in a short array or even the empty array. (This
|
|
behavior intentionally imitates Python.)
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr -n put '
|
|
end {
|
|
x = [10, 20, 30, 40, 50];
|
|
print x[1];
|
|
print x[5];
|
|
print x[6]; # absent
|
|
}
|
|
'
|
|
GENMD-EOF
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr -n put '
|
|
end {
|
|
x = [10, 20, 30, 40, 50];
|
|
print x[1:2];
|
|
print x[1:6];
|
|
print x[10:20];
|
|
}
|
|
'
|
|
GENMD-EOF
|
|
|
|
## Auto-create results in maps
|
|
|
|
As noted on the [maps page](reference-main-maps.md), indexing any
|
|
as-yet-assigned local variable or out-of-stream variable results in
|
|
**auto-create** of that variable as a map variable:
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr --csv --from example.csv put -q '
|
|
# You can do this but you do not need to:
|
|
# begin { @last_rates = {} }
|
|
@last_rates[$shape] = $rate;
|
|
end {
|
|
dump @last_rates;
|
|
}
|
|
'
|
|
GENMD-EOF
|
|
|
|
*This also means that auto-create results in maps, not arrays, even if keys are integers.*
|
|
If you want to auto-extend an [array](reference-main-arrays.md), initialize it explicitly to `[]`.
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr --csv --from example.csv head -n 4 then put -q '
|
|
begin {
|
|
@my_array = [];
|
|
}
|
|
@my_array[NR] = $quantity;
|
|
@my_map[NR] = $rate;
|
|
end {
|
|
dump
|
|
}
|
|
'
|
|
GENMD-EOF
|
|
|
|
## Auto-extend and null-gaps
|
|
|
|
Once an array is initialized, it can be extended by assigning to indices beyond
|
|
its length. If each write is one past the end of the array, the array will
|
|
grow by one. (Memory management, handled for you, is careful handled here in
|
|
Miller: not to worry, capacity is doubled so performance doesn't suffer a
|
|
rellocate on every single extend.)
|
|
|
|
This is important in Miller so you can do things like `@records[NR] = $*` with
|
|
a minimum of keystrokes without worrying about explicitly resizing arrays. In
|
|
particular, you can iteratively populate arrays as you read your data files,
|
|
without having to first know how many records they have.
|
|
|
|
However, if an array is written to more than one past its end, [values of type
|
|
JSON-null](reference-main-data-types.md) are used to fill in the gaps. These
|
|
are called **null-gaps**.
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr -n put '
|
|
end {
|
|
no_gaps = [];
|
|
no_gaps[1] = "a";
|
|
no_gaps[2] = "b";
|
|
|
|
gaps = [];
|
|
gaps[1] = "a";
|
|
gaps[5] = "e";
|
|
|
|
print no_gaps;
|
|
print gaps;
|
|
}
|
|
'
|
|
GENMD-EOF
|
|
|
|
## Unset as shift
|
|
|
|
Unsetting an array index results in shifting all higher-index elements down by one:
|
|
|
|
GENMD-RUN-COMMAND
|
|
mlr -n put '
|
|
end {
|
|
x = [ "a", "b", "c", "d", "e"];
|
|
print x;
|
|
unset x[2];
|
|
print x;
|
|
}
|
|
'
|
|
GENMD-EOF
|
|
|
|
More generally, you can get shift and pop operations by unsetting indices 1 and -1:
|
|
|
|
GENMD-CARDIFY-HIGHLIGHT-ONE
|
|
$ mlr repl -q
|
|
[mlr] x=[1,2,3,4,5]
|
|
[mlr] unset x[-1]
|
|
[mlr] x
|
|
[1, 2, 3, 4]
|
|
[mlr] unset x[-1]
|
|
[mlr] x
|
|
[1, 2, 3]
|
|
[mlr]
|
|
[mlr] x=[1,2,3,4,5]
|
|
[mlr] unset x[1]
|
|
[mlr] x
|
|
[2, 3, 4, 5]
|
|
[mlr] unset x[1]
|
|
[mlr] x
|
|
[3, 4, 5]
|
|
[mlr]
|
|
GENMD-EOF
|
|
|
|
## Looping
|
|
|
|
See [single-variable for-loops](reference-dsl-control-structures.md#single-variable-for-loops) and [key-value for-loops](reference-dsl-control-structures.md#key-value-for-loops).
|
|
|
|
## Array-valued fields in CSV files
|
|
|
|
See the [flatten/unflatten page](flatten-unflatten.md).
|