Skip to content

GH-551: [Java] Add TimeStamp*TZ support to ComplexCopier - #1414

Open
henriquejsza wants to merge 1 commit into
apache:mainfrom
henriquejsza:gh-551-complexcopier-timestamp-tz
Open

henriquejsza wants to merge 1 commit into
apache:mainfrom
henriquejsza:gh-551-complexcopier-timestamp-tz

Conversation

@henriquejsza

@henriquejsza henriquejsza commented Oct 30, 2026 •

Copy link
Copy Markdown

What's Changed

ComplexCopier skipped every minor type that has type parameters except
Decimal, so copying a list or struct containing TimeStampSecTZ,
TimeStampMilliTZ, TimeStampMicroTZ, or TimeStampNanoTZ threw
UnsupportedOperationException.

This generates the copy and the list/struct writer-lookup cases for the
timezone-aware timestamp types. Values are written with
write(TimeStamp*TZHolder) using the reader's timezone, so copying into an
empty target keeps the source timezone instead of falling back to UTC. In the
struct lookup, a reader whose field type is not a Timestamp (for example a
union-backed child) throws an UnsupportedOperationException with a clear
message. The fix is in the FreeMarker template
vector/src/main/codegen/templates/ComplexCopier.java.

Map keys and values of these types still throw: UnionMapWriter has no TZ
overrides for the key and value writers yet, so that is left for a follow-up.

Tests copy a list and a struct of TimeStampNanoTZ with a non-UTC zone into
empty targets and compare them with VectorEqualsVisitor, and check that the
map case and the union-backed struct case throw.

Closes #551.

@jbonofre jbonofre added the enhancement PRs that add or improve features. label Oct 31, 2026
@apache apache deleted a comment from github-actions Bot Oct 31, 2026
@github-actions github-actions Bot added this to the 20.0.0 milestone Oct 31, 2026
<#assign fields = minor.fields!type.fields />
<#assign uncappedName = name?uncap_first/>
<#if !minor.typeParams?? || minor.class?starts_with("Decimal") >
<#if !minor.typeParams?? || minor.class?starts_with("Decimal") || is_timestamp_tz(minor.class) >

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding the TZ types to the map lookup turns an UnsupportedOperationException into silently wrong data. UnionMapWriter only overrides the key/value getters for types without typeParams (plus decimal, decimal256 and fixedSizeBinary). So writer.value().timeStampNanoTZ() falls through to UnionListWriter.timeStampNanoTZ(), which returns the map writer itself. The value is then written into the map entries vector instead of the value field, and the entries vector gets promoted to a union.

Copying into MapVector.empty(...) would give the same result.

We could either:

  • drop the TZ case from getMapWriterForReader, so maps keep throwing and the list/struct support can land on its own, or
  • add TZ overrides to UnionMapWriter that route to the key/value fields of entryWriter and carry the timezone, as the struct path does?

mapWriter.startMap();
mapWriter.startEntry();
mapWriter.key().integer().writeInt(i);
mapWriter.value().timeStampNanoTZ().writeTimeStampNanoTZ(i * 1_000_000L);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test can't catch the problem mentioned above, because the source map is already malformed. mapWriter.value().timeStampNanoTZ() hits the same missing UnionMapWriter override, so from already has its entries promoted to a union and no value data. The comment at the top of the test blames the mismatch on timezones, but I think both sides are UTC here, the vectors differ in structure, not timezone.

Could we build the source directly on the vectors (startNewValue/endValue on the map, set key/value on the entries StructVector)? Then use a non-UTC zone and assert VectorEqualsVisitor.vectorEquals(from, to) instead of assertDoesNotThrow plus the value count. If we drop the map case instead, an assertThrows(UnsupportedOperationException.class, ...) would document that maps are not support yet 😄

@jbonofre jbonofre removed this from the 20.0.0 milestone Nov 16, 2026
ComplexCopier skipped every minor type that has type parameters except
Decimal, so copying a list or struct containing TimeStampSecTZ,
TimeStampMilliTZ, TimeStampMicroTZ, or TimeStampNanoTZ threw
UnsupportedOperationException.

Generate the copy and the list and struct writer-lookup cases for the
timezone-aware timestamp types, and write the values with their
timezone so the copy keeps it. Map keys and values of these types
still throw, since UnionMapWriter has no TZ overrides for them yet.

Closes apache#551.

Signed-off-by: henriquejsza <henriquejsza@gmail.com>
@henriquejsza
henriquejsza force-pushed the gh-551-complexcopier-timestamp-tz branch from baa67ea to 42541e0 Compare November 18, 2026 01:26
@henriquejsza

Copy link
Copy Markdown
Author

@jbonofre thanks for the review, you're right on both.

I dropped the TZ case from the map lookup, so maps keep throwing for now. The map test builds the source on the vectors and asserts the UnsupportedOperationException.

While at it I noticed the copy used the deprecated writeTimeStamp*TZ(long), which falls back to UTC when the target child doesn't exist yet. It now writes a TimeStamp*TZHolder with the reader's timezone, and the list/struct tests use America/Sao_Paulo with empty targets and vectorEquals. The struct lookup also throws a clear exception when the field type isn't a Timestamp (e.g. a union-backed child) instead of the NPE from timeStampNanoTZ(name).

Rebased on main to fix the conflict. Happy to open a follow-up issue for the UnionMapWriter TZ overrides.

@github-actions github-actions Bot added this to the 20.0.0 milestone Nov 18, 2026
@lidavidm lidavidm removed this from the 20.0.0 milestone Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement PRs that add or improve features.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ComplexCopier is missing cases for TimeStamp*TZ

3 participants