fromdate should support timezone offsets

[CFoltin]

Hi,

parsing the following command:
"2015-12-24T09:29:30+01:00" | fromdate
gives the error
jq: error (at :49): date "2015-12-24T09:29:30+01:00" does not match format "%Y-%m-%dT%H:%M:%SZ"
although it seems pretty ok for. Only timezone "Z" seems to work.

BR, Chris from FreeMind

[nicowilliams]

Supporting timezones in general is problematic. Currently we use the C library for dealing with parsing and formatting datetime strings. The more features of the C library we use, the harder it will be to stop using it later. Also, the C library's datetime capabilities vary quite a bit from one version/platform to another, so the more features we use, the more issues we can expect users to file that are ultimately caused by C library issues. Limiting datetime strings to UTC ISO8601 was an explicit choice to limit this pain. Adding timezone offset support when parsing is doable though, and probably worth the effort. I'm much less interested in adding timezone offsets when formatting datetime strings.

[CFoltin]

Hi,

thanks for the replay. AFAIK, the format I put to be parsed is a valid ISO8601 date. I've got the impression, that only the "timezone" 'Z' is working at jq, or could you give me an example of a different zone being parsed?

BR, Chris

[nicowilliams]

I wasn't clear. Only the "Z" zone is being parsed.

[nicowilliams]

One of the nice things about dates being in UTC is that then dates can be compared and ordered lexicographically. So I'm inclined to only support timezone offsets in fromdate.

Incidentally, you can use strptime() and the %z format specifier to parse ISO8061 timezones. Is that good enough for you?

[CFoltin]

Hi,
yes, an alternative would be ok.
I tried
"2015-12-24T09:29:30+01:00" | strptime("%Y-%m-%dT%H:%M:%SZ"), but same error.
Do you have an example at hand?
BR, Chris

[nicowilliams]

@CFoltin You should use "2015-12-24T09:29:30+01:00" | strptime("%Y-%m-%dT%H:%M:%S%z"), but note that glibc doesn't seem to support timezone offsets very well :(

(And, of course, %z is a glibc extension to the POSIX standard, so we can't rely on it.)

I see that glibc doesn't like a : in the tz offset, and it doesn't seem to adjust the result by the tz offset. I'm not inclined to work around this glibc bug, nor to implement timezone offsets in jq proper at this time, but I'll leave this open, and if someone submits a PR, we'll consider it!

[aliekens]

+1 for having timezone support in fromdate, even if that means working with UTC (or the computer's locale) internally and as todate output timezone.

Although dates aren't in the JSON standard, I see a bunch of use cases for jq processing. My current use case is to use group_by to organize objects in day or hour bins.

All my incoming JSON dates include the %z timezone format (Ruby's datetime) and I cannot change these inputs. I'm pondering if dates can be scanned and timezone-converted using sed or awk, before piping the JSON objects to jq, but that will be an aweful hack to work with.

@aliekens About your specific use case, since jq 1.5 has support for regular expressions and string substitutions and replacements, you can use this to rearrange the string so that the timezone appears where it can be parsed.

[aliekens]

Yay, I have figured out a way to correctly parse datetimes with timezones in jq, but it requires a bit of hacking. Here's how I can now parse my (Ruby's) datetimes:

$ TZ=/usr/share/zoneinfo/UTC jq -n '"2015-12-24T09:29:30+00:00" | sub("(?<before>.*):"; .before ) | strptime("%Y-%m-%dT%H:%M:%S%z") | todate'
"2015-12-24T09:29:30Z"
$ TZ=/usr/share/zoneinfo/UTC jq -n '"2015-12-24T09:29:30+01:00" | sub("(?<before>.*):"; .before ) | strptime("%Y-%m-%dT%H:%M:%S%z") | todate'
"2015-12-24T08:29:30Z"

Some notes:

• The example above is on a Mac. Behavior may be different in other environments because of C library differences. For example, jqplay (which runs on what platform?) always returns the same datetime, independent of the timezone.
• It is important to set jq's environment's timezone to UTC with TZ=/usr/share/zoneinfo/UTC or strptime will assume your computer locale's timezone is UTC (not good if your environment's locale is not UTC)
• strptime's %z format does not support "+01:00" timezones, it needs to be formatted as "+0100" (some implementations of strptime have a %: flag to support timezones with a colon). The last colon in the string is therefore subbed using a regex. (BTW, the docs need info or an example on how to use named captures)

[pkoppstein]

(BTW, the docs need info or an example on how to use named captures)

In the meantime, the FAQ has a question: Q: How are named capture variables used?