Repository navigation
Conversation
…fore taking it ArrowArrayStreamReader.loadNextBatch now releases the batch when the import fails before moving it out of the array, e.g. allocating the schema struct or the struct the batch moves into. Before, the struct was closed without calling its release callback, which leaked the producer's batch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
| @Test | ||
| public void streamImportOomDoesNotLeakExportedBatch() throws Exception { | ||
| // Reading the schema lazily, the first batch can fail allocating the schema struct. | ||
| streamImportOomDoesNotLeakExportedBatch(false); | ||
| // Reading the schema first, it can fail allocating the struct the batch moves into. | ||
| streamImportOomDoesNotLeakExportedBatch(true); | ||
| } |
There was a problem hiding this comment.
You should be able to do this with JUnit 5's test parameterization
There was a problem hiding this comment.
Done in dc15243: it is now one @ParameterizedTest with @ValueSource(booleans = {false, true}).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| try { | ||
| Data.importIntoVectorSchemaRoot(allocator, array, getVectorSchemaRoot(), provider); | ||
| } catch (Throwable t) { | ||
| // If the import failed before moving the batch out of the array, e.g. allocating the | ||
| // schema, the batch is still in the array: release it. | ||
| if (array.snapshot().release != NULL) { | ||
| array.release(); | ||
| } | ||
| throw t; | ||
| } |
There was a problem hiding this comment.
The PR wraps the 4-argument call:
try {
Data.importIntoVectorSchemaRoot(allocator, array, getVectorSchemaRoot(), provider);
} catch (Throwable t) {
if (array.snapshot().release != NULL) {
array.release();
}
throw t;
}The 4-argument Data.importIntoVectorSchemaRoot defaults to closeImportedStructs = true. Inside the 5-argument Data.importIntoVectorSchemaRoot:
- Line 487 calls
importIntoVector(..., closeImportedStructs)which callsarray.close(). - Line 488-492 then run
StructVectorUnloader.getRecordBatch()andVectorLoader.load(recordBatch).
If anything throws during unloading or loading (for example, an OutOfMemoryException in BaseVariableWidthVector.setReaderAndWriterIndex when allocating an offset buffer, or an IllegalArgumentException in StructVectorUnloader / VectorLoader), execution enters the catch (Throwable t) block while array is already closed. Calling array.snapshot() then hits checkNotNull(data, "ArrowArray is already closed") in ArrowArray.memoryAddress, throwing NullPointerException and masking the original exception t.
I suggest to pass closeImportedStructs = false since the enclosing try (ArrowArray array = ArrowArray.allocateNew(allocation)) already owns closing array (this also removes the redundant double-close() on the happy path):
| try { | |
| Data.importIntoVectorSchemaRoot(allocator, array, getVectorSchemaRoot(), provider); | |
| } catch (Throwable t) { | |
| // If the import failed before moving the batch out of the array, e.g. allocating the | |
| // schema, the batch is still in the array: release it. | |
| if (array.snapshot().release != NULL) { | |
| array.release(); | |
| } | |
| throw t; | |
| } | |
| try { | |
| Data.importIntoVectorSchemaRoot( | |
| allocator, array, getVectorSchemaRoot(), provider, /* closeImportedStructs= */ false); | |
| } catch (Throwable t) { | |
| // If the import failed before moving the batch out of the array, e.g. allocating the | |
| // schema, the batch is still in the array: release it. | |
| if (array.snapshot().release != NULL) { | |
| try { | |
| array.release(); | |
| } catch (Throwable suppressed) { | |
| t.addSuppressed(suppressed); | |
| } | |
| } | |
| throw t; | |
| } |
There was a problem hiding this comment.
Thanks, good catch. Applied in 2d037af: the import now passes closeImportedStructs = false, so the array stays open for the failure handling and the enclosing try-with-resources closes it once. A failure releasing the batch is added to the original exception as suppressed.
I didn't add a test for this path because I couldn't find an input that fails after the import closes the array. ArrayImporter.importBuffers loads the buffers into the vectors before the array is closed, including allocating a missing validity bitmap. After that, unloading the struct and loading the root only retain the imported buffers. setReaderAndWriterIndex allocates only when the offset buffer holds fewer than length + 1 offsets, and the importer rejects such a buffer. So the allocator-limit sweep in the stream test doesn't reach this path.
The 4-argument importIntoVectorSchemaRoot closes the array before unloading and loading the batch, so a failure there made the failure handling read a closed array: the NullPointerException masked the original exception. Import with closeImportedStructs = false and let the enclosing try-with-resources close the array, and add a failure to release the batch to the original exception as suppressed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What's Changed
ArrowArrayStreamReader.loadNextBatchnow releases the batch when the import fails before moving it out of the array: allocating the schema struct on the first batch, or allocating the struct the batch moves into. Before this change, the struct was closed without calling its release callback, and the producer's batch leaked. This is the stream counterpart of #1239, which covered failures while importing the buffers.The change and the test were written with AI assistance (Claude Code). I reproduced the leak and verified the fix locally.
Closes #1491.
Generated-by: Claude Code