From 2bca2d082bfc29b3839aa572d8619c819c19a7c9 Mon Sep 17 00:00:00 2001 From: John Kerl Date: Wed, 20 Oct 2021 23:48:43 -0400 Subject: [PATCH] Datetime/timezones docs page (#707) * xvright * Left/right trim on --repifs * Fix emit-flatten when main & emit output file formats differ * Port localtime DSL functions from C to Go * Port localtime DSL functions from C to Go (#703) * xvright * Left/right trim on --repifs * Fix emit-flatten when main & emit output file formats differ * Port localtime DSL functions from C to Go * Time-related issues: 331, 332, 359, 477 --- docs6/mkdocs.yml | 1 + docs6/src/record-heterogeneity.md | 2 - docs6/src/reference-dsl-time.md | 309 ++++++++++++++++++++++ docs6/src/reference-dsl-time.md.in | 226 ++++++++++++++++ docs6/src/reference-main-data-types.md | 1 + docs6/src/reference-main-data-types.md.in | 1 + go/todo.txt | 37 +-- 7 files changed, 540 insertions(+), 37 deletions(-) create mode 100644 docs6/src/reference-dsl-time.md create mode 100644 docs6/src/reference-dsl-time.md.in diff --git a/docs6/mkdocs.yml b/docs6/mkdocs.yml index 0a68ed38e..ba5578854 100644 --- a/docs6/mkdocs.yml +++ b/docs6/mkdocs.yml @@ -93,6 +93,7 @@ nav: - "DSL filter statements": "reference-dsl-filter-statements.md" - "DSL output statements": "reference-dsl-output-statements.md" - "DSL unset statements": "reference-dsl-unset-statements.md" + - "DSL datetime/timezone functions": "reference-dsl-time.md" - "DSL errors and transparency": "reference-dsl-errors.md" - "Differences from other programming languages": "reference-dsl-differences.md" - "A note on the complexity of Miller's expression language": "reference-dsl-complexity.md" diff --git a/docs6/src/record-heterogeneity.md b/docs6/src/record-heterogeneity.md index 752a70e46..3e5e1a3ad 100644 --- a/docs6/src/record-heterogeneity.md +++ b/docs6/src/record-heterogeneity.md @@ -127,8 +127,6 @@ If you `mlr csv cat` this, you'll get an error message: mlr --csv cat data/het/ragged.csv 
-a,b,c
-1,2,3
 mlr :  mlr: CSV header/data length mismatch 3 != 2 at filename data/het/ragged.csv row 3.
diff --git a/docs6/src/reference-dsl-time.md b/docs6/src/reference-dsl-time.md new file mode 100644 index 000000000..ed57405ba --- /dev/null +++ b/docs6/src/reference-dsl-time.md @@ -0,0 +1,309 @@ + +
+ +Quick links: +  +Flag list +  +Verb list +  +Function list +  +Glossary +  +Repository ↗ + +
+# DSL datetime/timezone functions + +Dates/times are not a separate data type; Miller uses ints for +[seconds since the epoch](https://en.wikipedia.org/wiki/Unix_time) and strings for formatted +date/times. In this page we take a look at what some of the various options are +for processing datetimes andd timezones in your data. + +See also the [section on time-related +functions](reference-dsl-builtin-functions/index.html#time-functions) for +information auto-generated from Miller's online-help strings. + +# Epoch seconds + +[Seconds since the epoch](https://en.wikipedia.org/wiki/Unix_time), or _Unix +Time_, is seconds (positive, zero, or negative) since midnight January 1 1970 +UTC. This representation has several advantages, and is quite common in the +computing world. + +Since this is a [number](reference-main-arithmetic.md) in Miller -- 64-bit +signed integer or double-precision floating-point -- this can represent dates +billions of years into the past or future without worry of overflow. (There is +no [year-2038 problem](https://en.wikipedia.org/wiki/Year_2038_problem) here.) +Being numbers, epoch-seconds are easy to store in databases, communicate over +networks in binary format, etc. Another benefit of epoch-seconds is that +they're independent of timezone or daylight-savings time. + +One minus is that, being just numbers, they're not particularly human-readable +-- hence the to-string and from-string functions described below. Another +caveat (not really a minus) is that _epoch milliseconds_ are common in some +contexts, particulary JavaScript. If you every see a timestamp for the year +49,000-something -- probablhy someone is treating epoch-milliseconds as +epoch-seconds. + +
+mlr -n put 'end {
+  print sec2gmt(1500000000);
+  print sec2gmt(1500000000000);
+}'
+
+
+2017-07-14T02:40:00Z
+49503-02-10T02:40:00Z
+
+ +You can get the current system time, as epoch-seconds, using the +[systime](reference-dsl-builtin-functions.md#systime) DSL function: + +
+mlr --c2p --from example.csv put '$t = systime()'
+
+
+color  shape    flag  k  index quantity rate   t
+yellow triangle true  1  11    43.6498  9.8870 1634784588.045347
+red    square   true  2  15    79.2778  0.0130 1634784588.045385
+red    circle   true  3  16    13.8103  2.9010 1634784588.045386
+red    square   false 4  48    77.5542  7.4670 1634784588.045393
+purple triangle false 5  51    81.2290  8.5910 1634784588.045394
+red    square   false 6  64    77.1991  9.5310 1634784588.045417
+purple triangle false 7  65    80.1405  5.8240 1634784588.045418
+yellow circle   true  8  73    63.9785  4.2370 1634784588.045419
+yellow circle   true  9  87    63.5058  8.3350 1634784588.045421
+purple square   false 10 91    72.3735  8.2430 1634784588.045422
+
+ +The [systimeint](reference-dsl-builtin-functions.md#systimeint) DSL functions +is nothing more than a keystroke-saver for `int(systime())`. + +# UTC times with standard format + +One way to make epoch-seconds human-readable, while maintaining some of their +benefits such as being independent of timezone and daylight savings, is to use +the [ISO8601](https://en.wikipedia.org/wiki/ISO_8601) format. This was the +first (and initially only) human-readable date/time format supported by Miller +going all the way back to Miller 1.0.0. + +You can get these from epoch-seconds using the +[sec2gmt](reference-dsl-builtin-functions.md#sec2gmt) DSL function. +(Note that the terms _UTC_ and _GMT_ are used interchangeably in Miller.) +We also have [sec2gmtdate](reference-dsl-builtin-functions.md#sec2gmtdate) DSL function. + +
+mlr -n put 'end {
+  print sec2gmt(0);
+  print sec2gmt(1234567890.123);
+  print sec2gmt(-1234567890.123);
+  print;
+  print sec2gmtdate(0);
+  print sec2gmtdate(1234567890.123);
+  print sec2gmtdate(-1234567890.123);
+}'
+
+
+1970-01-01T00:00:00Z
+2009-02-13T23:31:30Z
+1930-11-18T00:28:29Z
+
+1970-01-01
+2009-02-13
+1930-11-18
+
+ +# Local times with standard format; specifying timezones + +You can use similar formatting for dates in your preferred timezone, not just UTC/GMT. +We have the +[sec2localtime](reference-dsl-builtin-functions.md#sec2localtime), +[sec2localdate](reference-dsl-builtin-functions.md#sec2localdate), and +[localtime2sec](reference-dsl-builtin-functions.md#localtime2sec) DSL functions. + +You can specify the timezone using any of the following: + +* An environment variable, e.g. `export TZ=Asia/Istanbul` at your system prompt (`set TZ=Asia/Istanbul` in Windows) +* Using the `--tz` flag. This sets the `TZ` environment variable, but only internally to the `mlr` process +* Within a DSL expression, you can assign to `ENV["TZ"]`. + +Regardless, if you specifiy an invalid timezone, you'll be clearly notified: + +
+mlr --from example.csv --tz This/Is/A/Typo cat
+
+
+mlr: unknown time zone This/Is/A/Typo
+
+ +
+export TZ=Asia/Istanbul
+mlr -n put 'end { print sec2localtime(0) }'
+
+
+1970-01-01 02:00:00
+
+ +
+mlr --tz America/Sao_Paulo -n put 'end { print sec2localtime(0) }'
+
+
+1969-12-31 21:00:00
+
+ +
+mlr -n put 'end {
+  ENV["TZ"] = "Asia/Istanbul";
+  print sec2localtime(0);
+  print sec2localdate(0);
+  print localtime2sec("2000-01-02 03:04:05");
+  print;
+  ENV["TZ"] = "America/Sao_Paulo";
+  print sec2localtime(0);
+  print sec2localdate(0);
+  print localtime2sec("2000-01-02 03:04:05");
+}'
+
+
+1970-01-01 02:00:00
+1970-01-01
+946775045
+
+1969-12-31 21:00:00
+1969-12-31
+946789445
+
+ +Note that for local times, Miller omits the `T` and the `Z` you see in GMT times. + +We also have the +[gmt2localtime](reference-dsl-builtin-functions.md#gmt2localtime) and +[localtime2gmt](reference-dsl-builtin-functions.md#localtime2gmt) convenience functions: + +
+mlr -n put 'end {
+  ENV["TZ"] = "Asia/Istanbul";
+  print gmt2localtime("1970-01-01T00:00:00Z");
+  print localtime2gmt("1970-01-01 00:00:00");
+}'
+
+
+1970-01-01 02:00:00
+1969-12-31T22:00:00Z
+
+ +# GMT and local times with custom formats + +The to-string and from-string functions we've seen so far are low-keystroking: +with a little bit of typing you can convert datetimes to/from epoch seconds. +The minus, however, is flexibility. This is where the +[strftime](reference-dsl-builtin-functions.md#strftime), +[strptime](reference-dsl-builtin-functions.md#strptime) functions come into play. + +Notes: + +* The names `strftime` and `strptime` far predate Miller; they were chosen for familiarity. The `f` is for _format_: from epoch-seconds to human-readable string. The `p` is for _parse_: for doing the reverse. +* Even though Miller is written in Go as of Miller 6, it still preserves [C-like](https://en.wikipedia.org/wiki/C_date_and_time_functions#strftime) `strftime` and `strptime` semantics. + * For `strftime`, this is thanks to [https://github.com/lestrrat-go/strftime](https://github.com/lestrrat-go/strftime). + * For `stpftime`, this is thanks to [https://github.com/pbnjay/strptime](https://github.com/pbnjay/strptime). +* See [https://devhints.io/strftime](https://devhints.io/strftime) for sample format strings you can use. + +Some examples: + +
+mlr -n put 'end {
+  ENV["TZ"] = "Asia/Istanbul";
+  print strftime(0, "%Y-%m-%d %H:%M:%S");
+  print strftime(0, "%Y-%m-%d %H:%M:%S %Z");
+  print strftime(0, "%Y-%m-%d %H:%M:%S %z");
+  print strftime(0, "%A, %B %e, %Y");
+  print strftime(123456789, "%I:%M %p");
+}'
+
+
+1970-01-01 00:00:00
+1970-01-01 00:00:00 UTC
+1970-01-01 00:00:00 +0000
+Thursday, January  1, 1970
+09:33 PM
+
+ +Unfortunately, names from `%A` and `%B` are only available in English, as an +artifact of a design choice in the Go `time` library which Miller (and its +`strftime` / `strptime` supporting packages as noted above) rely on. + +We also have +[strftimelocal](reference-dsl-builtin-functions.md#strftimelocal) and +[strptimelocal](reference-dsl-builtin-functions.md#strptimelocal): + +
+mlr -n put 'end {
+  ENV["TZ"] = "America/Anchorage";
+  print strftime_local(0, "%Y-%m-%d %H:%M:%S %Z");
+  print strftime_local(0, "%Y-%m-%d %H:%M:%S %z");
+  print strftime_local(0, "%A, %B %e, %Y");
+  print strptime_local("2020-03-01 00:00:00", "%Y-%m-%d %H:%M:%S");
+  print;
+  ENV["TZ"] = "Asia/Hong_Kong";
+  print strftime_local(0, "%Y-%m-%d %H:%M:%S %Z");
+  print strftime_local(0, "%Y-%m-%d %H:%M:%S %z");
+  print strftime_local(0, "%A, %B %e, %Y");
+  print strptime_local("2020-03-01 00:00:00", "%Y-%m-%d %H:%M:%S");
+}'
+
+
+1969-12-31 14:00:00 AHST
+1969-12-31 14:00:00 -1000
+Wednesday, December 31, 1969
+1583053200
+
+1970-01-01 08:00:00 HKT
+1970-01-01 08:00:00 +0800
+Thursday, January  1, 1970
+1582992000
+
+ +# Relative times + +You can get the seconds since the Miller process start using +[uptime](reference-dsl-builtin-functions.md#uptime): + + +
+color  shape    flag  k  index quantity rate   u
+
+
+yellow triangle true  1  11    43.6498  9.8870 0.0011110305786132812
+red    square   true  2  15    79.2778  0.0130 0.0011241436004638672
+red    circle   true  3  16    13.8103  2.9010 0.0011250972747802734
+red    square   false 4  48    77.5542  7.4670 0.0011301040649414062
+purple triangle false 5  51    81.2290  8.5910 0.0011301040649414062
+red    square   false 6  64    77.1991  9.5310 0.002481222152709961
+purple triangle false 7  65    80.1405  5.8240 0.0024831295013427734
+yellow circle   true  8  73    63.9785  4.2370 0.0024831295013427734
+yellow circle   true  9  87    63.5058  8.3350 0.0024852752685546875
+purple square   false 10 91    72.3735  8.2430 0.002485990524291992
+
+ +Time-differences can be done in seconds, of course; you can also use the following if you like: + +
+mlr -F | grep hms
+
+
+dhms2fsec  (class=time #args=1) Recovers floating-point seconds as in dhms2fsec("5d18h53m20.250000s") = 500000.250000
+dhms2sec  (class=time #args=1) Recovers integer seconds as in dhms2sec("5d18h53m20s") = 500000
+fsec2dhms  (class=time #args=1) Formats floating-point seconds as in fsec2dhms(500000.25) = "5d18h53m20.250000s"
+fsec2hms  (class=time #args=1) Formats floating-point seconds as in fsec2hms(5000.25) = "01:23:20.250000"
+hms2fsec  (class=time #args=1) Recovers floating-point seconds as in hms2fsec("01:23:20.250000") = 5000.250000
+hms2sec  (class=time #args=1) Recovers integer seconds as in hms2sec("01:23:20") = 5000
+sec2dhms  (class=time #args=1) Formats integer seconds as in sec2dhms(500000) = "5d18h53m20s"
+sec2hms  (class=time #args=1) Formats integer seconds as in sec2hms(5000) = "01:23:20"
+
+ +# References + +* List of formatting characters for `strftime` and `strptime`: [https://devhints.io/strftime](https://devhints.io/strftime) +* List of valid timezone names: [https://en.wikipedia.org/wiki/List_of_tz_database_time_zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) diff --git a/docs6/src/reference-dsl-time.md.in b/docs6/src/reference-dsl-time.md.in new file mode 100644 index 000000000..4d1070ee2 --- /dev/null +++ b/docs6/src/reference-dsl-time.md.in @@ -0,0 +1,226 @@ +# DSL datetime/timezone functions + +Dates/times are not a separate data type; Miller uses ints for +[seconds since the epoch](https://en.wikipedia.org/wiki/Unix_time) and strings for formatted +date/times. In this page we take a look at what some of the various options are +for processing datetimes andd timezones in your data. + +See also the [section on time-related +functions](reference-dsl-builtin-functions/index.html#time-functions) for +information auto-generated from Miller's online-help strings. + +# Epoch seconds + +[Seconds since the epoch](https://en.wikipedia.org/wiki/Unix_time), or _Unix +Time_, is seconds (positive, zero, or negative) since midnight January 1 1970 +UTC. This representation has several advantages, and is quite common in the +computing world. + +Since this is a [number](reference-main-arithmetic.md) in Miller -- 64-bit +signed integer or double-precision floating-point -- this can represent dates +billions of years into the past or future without worry of overflow. (There is +no [year-2038 problem](https://en.wikipedia.org/wiki/Year_2038_problem) here.) +Being numbers, epoch-seconds are easy to store in databases, communicate over +networks in binary format, etc. Another benefit of epoch-seconds is that +they're independent of timezone or daylight-savings time. + +One minus is that, being just numbers, they're not particularly human-readable +-- hence the to-string and from-string functions described below. Another +caveat (not really a minus) is that _epoch milliseconds_ are common in some +contexts, particulary JavaScript. If you every see a timestamp for the year +49,000-something -- probablhy someone is treating epoch-milliseconds as +epoch-seconds. + +GENMD_RUN_COMMAND +mlr -n put 'end { + print sec2gmt(1500000000); + print sec2gmt(1500000000000); +}' +GENMD_EOF + +You can get the current system time, as epoch-seconds, using the +[systime](reference-dsl-builtin-functions.md#systime) DSL function: + +GENMD_CARDIFY_HIGHLIGHT_ONE +mlr --c2p --from example.csv put '$t = systime()' +color shape flag k index quantity rate t +yellow triangle true 1 11 43.6498 9.8870 1634784588.045347 +red square true 2 15 79.2778 0.0130 1634784588.045385 +red circle true 3 16 13.8103 2.9010 1634784588.045386 +red square false 4 48 77.5542 7.4670 1634784588.045393 +purple triangle false 5 51 81.2290 8.5910 1634784588.045394 +red square false 6 64 77.1991 9.5310 1634784588.045417 +purple triangle false 7 65 80.1405 5.8240 1634784588.045418 +yellow circle true 8 73 63.9785 4.2370 1634784588.045419 +yellow circle true 9 87 63.5058 8.3350 1634784588.045421 +purple square false 10 91 72.3735 8.2430 1634784588.045422 +GENMD_EOF + +The [systimeint](reference-dsl-builtin-functions.md#systimeint) DSL functions +is nothing more than a keystroke-saver for `int(systime())`. + +# UTC times with standard format + +One way to make epoch-seconds human-readable, while maintaining some of their +benefits such as being independent of timezone and daylight savings, is to use +the [ISO8601](https://en.wikipedia.org/wiki/ISO_8601) format. This was the +first (and initially only) human-readable date/time format supported by Miller +going all the way back to Miller 1.0.0. + +You can get these from epoch-seconds using the +[sec2gmt](reference-dsl-builtin-functions.md#sec2gmt) DSL function. +(Note that the terms _UTC_ and _GMT_ are used interchangeably in Miller.) +We also have [sec2gmtdate](reference-dsl-builtin-functions.md#sec2gmtdate) DSL function. + +GENMD_RUN_COMMAND +mlr -n put 'end { + print sec2gmt(0); + print sec2gmt(1234567890.123); + print sec2gmt(-1234567890.123); + print; + print sec2gmtdate(0); + print sec2gmtdate(1234567890.123); + print sec2gmtdate(-1234567890.123); +}' +GENMD_EOF + +# Local times with standard format; specifying timezones + +You can use similar formatting for dates in your preferred timezone, not just UTC/GMT. +We have the +[sec2localtime](reference-dsl-builtin-functions.md#sec2localtime), +[sec2localdate](reference-dsl-builtin-functions.md#sec2localdate), and +[localtime2sec](reference-dsl-builtin-functions.md#localtime2sec) DSL functions. + +You can specify the timezone using any of the following: + +* An environment variable, e.g. `export TZ=Asia/Istanbul` at your system prompt (`set TZ=Asia/Istanbul` in Windows) +* Using the `--tz` flag. This sets the `TZ` environment variable, but only internally to the `mlr` process +* Within a DSL expression, you can assign to `ENV["TZ"]`. + +Regardless, if you specifiy an invalid timezone, you'll be clearly notified: + +GENMD_RUN_COMMAND_TOLERATING_ERROR +mlr --from example.csv --tz This/Is/A/Typo cat +GENMD_EOF + +GENMD_RUN_COMMAND +export TZ=Asia/Istanbul +mlr -n put 'end { print sec2localtime(0) }' +GENMD_EOF + +GENMD_RUN_COMMAND +mlr --tz America/Sao_Paulo -n put 'end { print sec2localtime(0) }' +GENMD_EOF + +GENMD_RUN_COMMAND +mlr -n put 'end { + ENV["TZ"] = "Asia/Istanbul"; + print sec2localtime(0); + print sec2localdate(0); + print localtime2sec("2000-01-02 03:04:05"); + print; + ENV["TZ"] = "America/Sao_Paulo"; + print sec2localtime(0); + print sec2localdate(0); + print localtime2sec("2000-01-02 03:04:05"); +}' +GENMD_EOF + +Note that for local times, Miller omits the `T` and the `Z` you see in GMT times. + +We also have the +[gmt2localtime](reference-dsl-builtin-functions.md#gmt2localtime) and +[localtime2gmt](reference-dsl-builtin-functions.md#localtime2gmt) convenience functions: + +GENMD_RUN_COMMAND +mlr -n put 'end { + ENV["TZ"] = "Asia/Istanbul"; + print gmt2localtime("1970-01-01T00:00:00Z"); + print localtime2gmt("1970-01-01 00:00:00"); +}' +GENMD_EOF + +# GMT and local times with custom formats + +The to-string and from-string functions we've seen so far are low-keystroking: +with a little bit of typing you can convert datetimes to/from epoch seconds. +The minus, however, is flexibility. This is where the +[strftime](reference-dsl-builtin-functions.md#strftime), +[strptime](reference-dsl-builtin-functions.md#strptime) functions come into play. + +Notes: + +* The names `strftime` and `strptime` far predate Miller; they were chosen for familiarity. The `f` is for _format_: from epoch-seconds to human-readable string. The `p` is for _parse_: for doing the reverse. +* Even though Miller is written in Go as of Miller 6, it still preserves [C-like](https://en.wikipedia.org/wiki/C_date_and_time_functions#strftime) `strftime` and `strptime` semantics. + * For `strftime`, this is thanks to [https://github.com/lestrrat-go/strftime](https://github.com/lestrrat-go/strftime). + * For `stpftime`, this is thanks to [https://github.com/pbnjay/strptime](https://github.com/pbnjay/strptime). +* See [https://devhints.io/strftime](https://devhints.io/strftime) for sample format strings you can use. + +Some examples: + +GENMD_RUN_COMMAND +mlr -n put 'end { + ENV["TZ"] = "Asia/Istanbul"; + print strftime(0, "%Y-%m-%d %H:%M:%S"); + print strftime(0, "%Y-%m-%d %H:%M:%S %Z"); + print strftime(0, "%Y-%m-%d %H:%M:%S %z"); + print strftime(0, "%A, %B %e, %Y"); + print strftime(123456789, "%I:%M %p"); +}' +GENMD_EOF + +Unfortunately, names from `%A` and `%B` are only available in English, as an +artifact of a design choice in the Go `time` library which Miller (and its +`strftime` / `strptime` supporting packages as noted above) rely on. + +We also have +[strftimelocal](reference-dsl-builtin-functions.md#strftimelocal) and +[strptimelocal](reference-dsl-builtin-functions.md#strptimelocal): + +GENMD_RUN_COMMAND +mlr -n put 'end { + ENV["TZ"] = "America/Anchorage"; + print strftime_local(0, "%Y-%m-%d %H:%M:%S %Z"); + print strftime_local(0, "%Y-%m-%d %H:%M:%S %z"); + print strftime_local(0, "%A, %B %e, %Y"); + print strptime_local("2020-03-01 00:00:00", "%Y-%m-%d %H:%M:%S"); + print; + ENV["TZ"] = "Asia/Hong_Kong"; + print strftime_local(0, "%Y-%m-%d %H:%M:%S %Z"); + print strftime_local(0, "%Y-%m-%d %H:%M:%S %z"); + print strftime_local(0, "%A, %B %e, %Y"); + print strptime_local("2020-03-01 00:00:00", "%Y-%m-%d %H:%M:%S"); +}' +GENMD_EOF + +# Relative times + +You can get the seconds since the Miller process start using +[uptime](reference-dsl-builtin-functions.md#uptime): + + +GENMD_CARDIFY_HIGHLIGHT_ONE +color shape flag k index quantity rate u +yellow triangle true 1 11 43.6498 9.8870 0.0011110305786132812 +red square true 2 15 79.2778 0.0130 0.0011241436004638672 +red circle true 3 16 13.8103 2.9010 0.0011250972747802734 +red square false 4 48 77.5542 7.4670 0.0011301040649414062 +purple triangle false 5 51 81.2290 8.5910 0.0011301040649414062 +red square false 6 64 77.1991 9.5310 0.002481222152709961 +purple triangle false 7 65 80.1405 5.8240 0.0024831295013427734 +yellow circle true 8 73 63.9785 4.2370 0.0024831295013427734 +yellow circle true 9 87 63.5058 8.3350 0.0024852752685546875 +purple square false 10 91 72.3735 8.2430 0.002485990524291992 +GENMD_EOF + +Time-differences can be done in seconds, of course; you can also use the following if you like: + +GENMD_RUN_COMMAND +mlr -F | grep hms +GENMD_EOF + +# References + +* List of formatting characters for `strftime` and `strptime`: [https://devhints.io/strftime](https://devhints.io/strftime) +* List of valid timezone names: [https://en.wikipedia.org/wiki/List_of_tz_database_time_zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) diff --git a/docs6/src/reference-main-data-types.md b/docs6/src/reference-main-data-types.md index 80d77e82d..1bb0cc7af 100644 --- a/docs6/src/reference-main-data-types.md +++ b/docs6/src/reference-main-data-types.md @@ -23,6 +23,7 @@ Miller's types are: * Scalars: * **string**: such as `"abcdefg"`, supporting concatenation, one-up indexing and slicing, and [library functions](reference-dsl-builtin-functions.md#string-functions). See the pages on [strings](reference-main-strings.md) and [regular expressions](reference-main-regular-expressions.md). * **float** and **int**: such as `1.2` and `3`: double-precision and 64-bit signed, respectively. See the section on [arithmetic operators and math-related library functions](reference-dsl-builtin-functions.md#math-functions) as well as the [Arithmetic](reference-main-arithmetic.md) page. + * dates/times are not a separate data type; Miller uses ints for [seconds since the epoch](https://en.wikipedia.org/wiki/Unix_time) and strings for formatted date/times. See the [DSL datetime/timezone functions page](reference-dsl-time.md) for more information. * **boolean**: literals `true` and `false`; results of `==`, `<`, `>`, etc. See the section on [boolean operators](reference-dsl-builtin-functions.md#boolean-functions). * Collections: * **map**: such as `{"a":1,"b":[2,3,4]}`, supporting key-indexing, preservation of insertion order, [library functions](reference-dsl-builtin-functions.md#collections-functions), etc. See the [Maps](reference-main-maps.md) page. diff --git a/docs6/src/reference-main-data-types.md.in b/docs6/src/reference-main-data-types.md.in index 2dcef0f87..0cbfc949e 100644 --- a/docs6/src/reference-main-data-types.md.in +++ b/docs6/src/reference-main-data-types.md.in @@ -7,6 +7,7 @@ Miller's types are: * Scalars: * **string**: such as `"abcdefg"`, supporting concatenation, one-up indexing and slicing, and [library functions](reference-dsl-builtin-functions.md#string-functions). See the pages on [strings](reference-main-strings.md) and [regular expressions](reference-main-regular-expressions.md). * **float** and **int**: such as `1.2` and `3`: double-precision and 64-bit signed, respectively. See the section on [arithmetic operators and math-related library functions](reference-dsl-builtin-functions.md#math-functions) as well as the [Arithmetic](reference-main-arithmetic.md) page. + * dates/times are not a separate data type; Miller uses ints for [seconds since the epoch](https://en.wikipedia.org/wiki/Unix_time) and strings for formatted date/times. See the [DSL datetime/timezone functions page](reference-dsl-time.md) for more information. * **boolean**: literals `true` and `false`; results of `==`, `<`, `>`, etc. See the section on [boolean operators](reference-dsl-builtin-functions.md#boolean-functions). * Collections: * **map**: such as `{"a":1,"b":[2,3,4]}`, supporting key-indexing, preservation of insertion order, [library functions](reference-dsl-builtin-functions.md#collections-functions), etc. See the [Maps](reference-main-maps.md) page. diff --git a/go/todo.txt b/go/todo.txt index a89404134..b4c6ca789 100644 --- a/go/todo.txt +++ b/go/todo.txt @@ -56,6 +56,8 @@ d pre-release: o draft release w/ m6 binaries YYYYMMDD -- ? o gha download steps clearer +* wut h1 spacing before/after ... + ---------------------------------------------------------------- LOCALTIME @@ -74,44 +76,9 @@ LOCALTIME * new webdoc separate PR - incl https://github.com/johnkerl/miller/issues/331 -* ENV["TZ"] update: - o --tz is a setenv & is treated the same. also ENV["TZ"] - o no explicit tz arg @ dsl functions, sorry ... - o now maybe simplify the RTs ... - o at very least, at a multi-tz case - o sec2gmt sec2localtime - sec2gmtdate sec2localdate - strftime strftime_local - o gmt2sec localtime2sec - strptime strptime_local - o gmt2localtime localtime2gmt - o note in new webdoc - mlr> sec2localtime(0) - "1969-12-31 19:00:00" - mlr> ENV["TZ"]="" - mlr> sec2localtime(0) - "1970-01-01 00:00:00" - mlr> ENV["TZ"]="GMT" - mlr> sec2localtime(0) - "1970-01-01 00:00:00" - mlr> ENV["TZ"]="Asia/Istanbul" - mlr> sec2localtime(0) - "1970-01-01 02:00:00" - mlr> ENV["TZ"]="America/Sao_Paulo" - mlr> sec2localtime(0) - "1969-12-31 21:00:00" - o ref: - https://www.gnu.org/software/libc/manual/html_node/TZ-Variable.html - http://www.iana.org/time-zones - https://en.wikipedia.org/wiki/Tz_database - https://en.wikipedia.org/wiki/List_of_tz_database_time_zones - * https://github.com/johnkerl/miller/issues/332 o localtime2gmt and gmt2localtime --tz - k maybe health-check with `location, err := time.LoadLocation("America/Los_Angeles")` -- ? - ? make this this applies to all datetimes that don't include a time zone - ? also make sure that, absent this, all zoneless datatimes are interpreted in that timezone ---------------------------------------------------------------- 398