Repository navigation
GH-551: [Java] Add TimeStamp*TZ support to ComplexCopier - #1414
henriquejsza wants to merge 1 commit into
Conversation
| <#assign fields = minor.fields!type.fields /> | ||
| <#assign uncappedName = name?uncap_first/> | ||
| <#if !minor.typeParams?? || minor.class?starts_with("Decimal") > | ||
| <#if !minor.typeParams?? || minor.class?starts_with("Decimal") || is_timestamp_tz(minor.class) > |
There was a problem hiding this comment.
Adding the TZ types to the map lookup turns an UnsupportedOperationException into silently wrong data. UnionMapWriter only overrides the key/value getters for types without typeParams (plus decimal, decimal256 and fixedSizeBinary). So writer.value().timeStampNanoTZ() falls through to UnionListWriter.timeStampNanoTZ(), which returns the map writer itself. The value is then written into the map entries vector instead of the value field, and the entries vector gets promoted to a union.
Copying into MapVector.empty(...) would give the same result.
We could either:
- drop the TZ case from
getMapWriterForReader, so maps keep throwing and the list/struct support can land on its own, or - add TZ overrides to
UnionMapWriterthat route to thekey/valuefields ofentryWriterand carry the timezone, as the struct path does?
| mapWriter.startMap(); | ||
| mapWriter.startEntry(); | ||
| mapWriter.key().integer().writeInt(i); | ||
| mapWriter.value().timeStampNanoTZ().writeTimeStampNanoTZ(i * 1_000_000L); |
There was a problem hiding this comment.
This test can't catch the problem mentioned above, because the source map is already malformed. mapWriter.value().timeStampNanoTZ() hits the same missing UnionMapWriter override, so from already has its entries promoted to a union and no value data. The comment at the top of the test blames the mismatch on timezones, but I think both sides are UTC here, the vectors differ in structure, not timezone.
Could we build the source directly on the vectors (startNewValue/endValue on the map, set key/value on the entries StructVector)? Then use a non-UTC zone and assert VectorEqualsVisitor.vectorEquals(from, to) instead of assertDoesNotThrow plus the value count. If we drop the map case instead, an assertThrows(UnsupportedOperationException.class, ...) would document that maps are not support yet 😄
ComplexCopier skipped every minor type that has type parameters except Decimal, so copying a list or struct containing TimeStampSecTZ, TimeStampMilliTZ, TimeStampMicroTZ, or TimeStampNanoTZ threw UnsupportedOperationException. Generate the copy and the list and struct writer-lookup cases for the timezone-aware timestamp types, and write the values with their timezone so the copy keeps it. Map keys and values of these types still throw, since UnionMapWriter has no TZ overrides for them yet. Closes apache#551. Signed-off-by: henriquejsza <henriquejsza@gmail.com>
baa67ea to
42541e0
Compare
|
@jbonofre thanks for the review, you're right on both. I dropped the TZ case from the map lookup, so maps keep throwing for now. The map test builds the source on the vectors and asserts the While at it I noticed the copy used the deprecated Rebased on main to fix the conflict. Happy to open a follow-up issue for the |
What's Changed
ComplexCopierskipped every minor type that has type parameters exceptDecimal, so copying a list or struct containingTimeStampSecTZ,TimeStampMilliTZ,TimeStampMicroTZ, orTimeStampNanoTZthrewUnsupportedOperationException.This generates the copy and the list/struct writer-lookup cases for the
timezone-aware timestamp types. Values are written with
write(TimeStamp*TZHolder)using the reader's timezone, so copying into anempty target keeps the source timezone instead of falling back to UTC. In the
struct lookup, a reader whose field type is not a
Timestamp(for example aunion-backed child) throws an
UnsupportedOperationExceptionwith a clearmessage. The fix is in the FreeMarker template
vector/src/main/codegen/templates/ComplexCopier.java.Map keys and values of these types still throw:
UnionMapWriterhas no TZoverrides for the key and value writers yet, so that is left for a follow-up.
Tests copy a list and a struct of
TimeStampNanoTZwith a non-UTC zone intoempty targets and compare them with
VectorEqualsVisitor, and check that themap case and the union-backed struct case throw.
Closes #551.