You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add support for scalar values with extension types #1301
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Suppose I have a pyarrow scalar value that contains an extension type. If I try turning that into a literal expression in datafusion, we should get the associated metadata transparently to the user.
This currently fails with ArrowTypeError: Expected bytes, got a 'UUID' object. That can be overcome with the simple patch
--- a/src/pyarrow_util.rs+++ b/src/pyarrow_util.rs@@ -30,7 +30,11 @@ impl FromPyArrow for PyScalarValue {
fn from_pyarrow_bound(value: &Bound<'_, PyAny>) -> PyResult<Self> {
let py = value.py();
let typ = value.getattr("type")?;
- let val = value.call_method0("as_py")?;+ let val = if value.hasattr("value")? {+ value.getattr("value")?+ } else {+ value.call_method0("as_py")?+ };
But then we still don't have the metadata. It is lost and we get a bare fixed sized binary.
Describe the solution you'd like
The above code should just work. I have done a little investigation and using the pycapsule interface we can get the schema of the array we generate inside PyScalarValue::from_pyarrow_bound. We can then plumb this through when calling lit().
Ideally we would take this opportunity to ensure that when we call PyScalarValue::from_pyarrow_bound we are also supporting other libraries besides just pyarrow. There has been a complaint a few times that we are too tightly coupled to pyarrow. In particular it would be good to demonstrate that when converting a Python object that is a scalar value it works for:
pyarrow
nanoarrow
arro3
polars
I don't think we necessarily need to support pandas since they are not an Arrow library.
Describe alternatives you've considered
Alternatively the user can manually turn their data into the underlying storage and then attach the metadata from their extension type. This feels like a poor user experience.
Additional context
This came up during a different investigation:
Also worth evaluating while we're doing this: For scalar values, is it possible for them to contain metadata? If I do pa.scalar(uuid.uuid4().bytes, type=pa.uuid()) and I check the type I should have the extension data. Maybe this is already supported, but as part of this PR I want to evaluate that as well.
Activity
zaryab-ali commentedon Feb 14, 2026
hi @timsaucer , if you are not working on this issue, can you assign it to me please
timsaucer commentedon Feb 14, 2026
This might be resolved in kosiew#7 which will probably merge soon
timsaucer commentedon Feb 21, 2026
It looks like the above code does not include the metadata support.