Repository navigation
GH-200: Test a second transfer for every vector kind - #1381
Conversation
This comment has been minimized.
This comment has been minimized.
|
The only failing check is "Ensure PR format", which needs a label. I don't have permission to add one; this is a |
There was a problem hiding this comment.
I think we should parameterize this test and test all types, ideally, instead of playing whack-a-mole. Or do other vectors have equivalent tests? (If so, I wonder if we could try and factor out common behavioral tests for vectors, instead of scattering them across different test classes)
There was a problem hiding this comment.
Good idea. Done in f41e9a8: I replaced the two DenseUnionVector-only tests with a parameterized test in TestSplitAndTransfer, where the transfer tests for the other vector kinds already live. For 18 kinds, it transfers a populated vector (or splits and transfers it), transfers the result again, and compares it with the original using VectorEqualsVisitor. The kinds are the fixed-width, variable-width and view vectors, fixed-size binary, the list, list-view and fixed-size list vectors, map, struct, union and dense union.
To answer the other question: without the fix, only the DenseUnionVector cases fail (expected [5, true, 7] but was [null, null, null]). The other 17 kinds already keep their values through a second transfer, so DenseUnionVector was the only one missing state there.
Factoring the other per-vector behavioural tests into shared parameterized tests looks worthwhile, but it touches many test classes, so I've kept it out of this PR. I can open an issue for it if that's useful.
| to.childVectors[i] = dstVec; | ||
| internalTransferPairs[i] = srcVec.makeTransferPair(dstVec); | ||
| } | ||
| to.nextTypeId = nextTypeId; |
There was a problem hiding this comment.
This cause a regression for unions whose type ids aren't 0..n-1, for example [5, 10], which is what IPC or the C Data interface gives us when the schema declares type ids.
createTransferPairs fills to.childVectors at the loop index (0 and 1), but setChildVectorValueCounts reads childVectors[typeMapFields[i]], i.e. childVectors[5] and childVectors[10], which are null on the target. Before this change to.nextTypeId was 0, so that loop never ran. Now any setValueCount on the target after transfer() (including through VectorSchemaRoot.setRowCount) throws:
java.lang.NullPointerException: Cannot invoke "org.apache.arrow.vector.ValueVector.setValueCount(int)" because "this.childVectors[...]" is null
at org.apache.arrow.vector.complex.DenseUnionVector.setChildVectorValueCounts(DenseUnionVector.java)
at org.apache.arrow.vector.complex.DenseUnionVector.setValueCount(DenseUnionVector.java)
Here's an example to reproduce the regression:
Field field =
new Field(
"u",
FieldType.nullable(new ArrowType.Union(UnionMode.Dense, new int[] {5, 10})),
asList(
Field.nullable("i", MinorType.INT.getType()),
Field.nullable("b", MinorType.BIT.getType())));
try (DenseUnionVector vector = (DenseUnionVector) field.createVector(allocator)) {
vector.allocateNew();
vector.setTypeId(0, (byte) 5);
vector.setSafe(0, intHolder(5));
vector.setTypeId(1, (byte) 10);
vector.setSafe(1, bitHolder(true));
vector.setTypeId(2, (byte) 5);
vector.setSafe(2, intHolder(7));
vector.setValueCount(3);
TransferPair pair = vector.getTransferPair(allocator);
try (DenseUnionVector target = (DenseUnionVector) pair.getTo()) {
pair.transfer();
target.setValueCount(3); // NPE
}
}There was a problem hiding this comment.
Thanks for the reproducer. The transfer pair indexed typeFields and childVectors by registration order instead of by type id, which stayed hidden while the target's nextTypeId was 0. 7f8447e maps them through typeMapFields in createTransferPairs, transfer() and splitAndTransfer(); the last one also sized its per-type arrays by the type count, so it overflowed for id 10. Your example is now testTransferWithDeclaredTypeIds, with a second transfer and a splitAndTransfer variant.
…onVector DenseUnionVector's transfer pair copied the type fields and type map into the target vector but left the target's nextTypeId at zero. The target therefore behaved as if no types were registered: transferring it again created no child transfer pairs and produced a vector whose values read back as null. Copy nextTypeId alongside the other type metadata when the transfer pairs are created, so both transfer() and splitAndTransfer() leave the target with the same registered types as the source. Closes apache#200.
Replace the two DenseUnionVector-only tests with a parameterized test in TestSplitAndTransfer. For 18 vector kinds it transfers, or splits and transfers, a populated vector, transfers the result again, and compares it with the original using VectorEqualsVisitor. Without the fix only the DenseUnionVector cases fail.
DenseUnionVector keeps typeFields and childVectors by type id and maps the i-th registered type to its id through typeMapFields, but the transfer pair indexed them by i. With declared type ids such as [5, 10], the target's setValueCount then failed on a null child, and splitAndTransfer indexed per-type arrays of length nextTypeId by type id.
7f8447e to
203536c
Compare
jbonofre
left a comment
There was a problem hiding this comment.
Thanks @sharanggupta!
I rebased on main. #1419 has already fixed the transfer code in DenseUnionVector (it copies nextTypeId and indexes the children by type id), and #1472 then changed how the children are paired. So I kept main version of DenseUnionVector.java, and the PR now adds only tests. They are still worth having: nothing on main transfers a vector a second time, which is what #200 reported, and the parameterized test in TestSplitAndTransfer covers that for 18 vector kinds.
Remove the trailing blank line before the closing brace flagged by spotless:check on JDK 21+.
What's Changed
#1419 has since fixed the
DenseUnionVectortransfer that #200 reported, so after the rebase onmainthis PR only adds tests for it.A parameterized test in
TestSplitAndTransfercovers 18 vector kinds. For each, it transfers a populated vector (or splits and transfers it), transfers the result again, and compares it with the original usingVectorEqualsVisitor.TestDenseUnionVectoradds a transfer-twice test and a split-and-transfer test for a dense union with declared type ids 5 and 10.Closes #200.