Repository navigation
Make scalar operands of binary elementwise functions asynchronous - #3182
ndgrigorian wants to merge 3 commits into
Conversation
A Python or NumPy scalar was turned into a 0-d array, or assigned through basic indexing, with _copy_numpy_ndarray_into_usm_ndarray. That copy waits for all work submitted to the queue, so every binary operation with a scalar operand (2 * x, x == 0), clip, where and x[0] = 1 blocked the host until the device was idle (gh-3148). _copy_from_numpy_into and asarray now fill 0-d input with _full_usm_ndarray. The value is cast on the host the way the copy kernel casts it: through float32/complex64 on devices without fp64, the real part of complex values, float-to-integer through int64 so that it wraps, and without NumPy's cast warnings.
|
View rendered docs @ https://intelpython.github.io/dpnp/pull/3182/index.html |
|
Array API standard conformance tests for dpnp=0.22.0dev1=np2py314h8d9cdd5_9 ran successfully. |
|
@ndgrigorian, would it make sense to extend ASV scope with a benchmark on scalar operands in binary elementwise functions? I guess we not need to cover all elementwise functions, but few or couple of most valuable ones. |
There was a problem hiding this comment.
🔵 Needs a closer look
Replacing SYCL casts with host NumPy casts still needs device-level validation for NaN, infinity, and out-of-int64-range values, including on fp64-less devices not covered by the reported runs.
No open issues 🎉
What changed in this PR
This PR addresses #3148 by replacing synchronizing host copies of scalar operands with asynchronous device fills in dpnp’s tensor layer, allowing scalar operations to return without waiting for queued device work.
Changes:
- Uses device fills for scalar and 0-d NumPy inputs in array creation and basic indexed assignment.
- Adds host-side casting intended to preserve copy-kernel conversion behavior.
- Adds regression tests for scalar creation, assignment, and binary operations, and documents the change.
| File | Description |
|---|---|
dpnp/tests/tensor/test_usm_ndarray_ctor.py |
Tests scalar assignment without host copies and checks casting behavior. |
dpnp/tests/tensor/test_tensor_asarray.py |
Tests scalar creation, dtype conversion, and devices without fp64 support. |
dpnp/tests/tensor/elementwise/test_elementwise_classes.py |
Tests scalar binary operands, including reversed, output, and in-place operations. |
dpnp/tensor/_ctors.py |
Routes 0-d NumPy inputs through device fills. |
dpnp/tensor/_copy_utils.py |
Adds scalar casting and asynchronous fills with event tracking. |
CHANGELOG.md |
Documents the removal of blocking scalar host copies. |
Review effort: Balanced
| if np_ary.ndim == 0: | ||
| # fill instead of the host copy, which waits for the whole queue | ||
| _manager = SequentialOrderManager[dst.sycl_queue] | ||
| _fill_from_numpy_scalar(dst, np_ary, _manager.submitted_events) | ||
| return |
There was a problem hiding this comment.
x[...] = 1 now returns before the fill runs. test_ctor_buffer_kwarg (test_usm_ndarray_ctor.py:533-537) reads the memory with usm_data.copy_to_host(), which does not wait for the fill on an out-of-order queue. On the OpenCL CPU (what CI runs) the test fails in 186 of 2000 runs; on master it never fails:
import numpy as np
import dpnp.tensor as dpt
stale = 0
for _ in range(2000):
x = dpt.usm_ndarray((10, 5), dtype="c8")
x[...] = 1
h = np.frombuffer(x.usm_data.copy_to_host(), dtype="c8")
stale += not (h == 1).all()
print(f"stale in {stale}/2000 runs") # CPU: 210/2000, GPU: 0/2000; master: 0/2000Could the test call Xusm.sycl_queue.wait() before copy_to_host()?
| fill_val = np_ary.astype(dst_dt)[()].item() | ||
| hev, fill_ev = ti._full_usm_ndarray( | ||
| fill_val, dst, sycl_queue=copy_q, depends=depends | ||
| ) | ||
| SequentialOrderManager[copy_q].add_event_pair(hev, fill_ev) |
There was a problem hiding this comment.
Assigning a scalar to a strided view is slower on GPU now: the strided _full_usm_ndarray path packs the shape into a device allocation and adds two host tasks per call, where the old blocking copy added none:
import time, dpnp
z = dpnp.zeros((2048, 2048), dtype="f4")
def fill_columns():
for i in range(200):
z[:, i] = 1.0
z.sycl_queue.wait()
fill_columns()
times = []
for _ in range(10):
t = time.perf_counter()
fill_columns()
times.append((time.perf_counter() - t) * 1e3)
print(f"{sorted(times)[5]:.1f} ms") # GPU Max 1100: 24 ms; master: 11 ms (CPU: 35 ms on both)After other work on the queue it gets worse (96 ms vs 11 ms in a longer run). Is this expected, or could the strided fill avoid the extra host tasks?
| np_ary = np_ary.real | ||
| # the copy kernel casts NaN and out-of-range values without warning | ||
| with np.errstate(all="ignore"): | ||
| if np_ary.dtype.kind == "f" and dst_dt.kind in "iu": | ||
| # the copy kernel narrows through int64, so it wraps like NumPy | ||
| # integer casts where a direct NumPy float cast saturates | ||
| if dst_dt.itemsize < 8 or (dst_dt.kind == "u" and np_ary < 0): | ||
| np_ary = np_ary.astype(np.int64) | ||
| fill_val = np_ary.astype(dst_dt)[()].item() | ||
| hev, fill_ev = ti._full_usm_ndarray( |
There was a problem hiding this comment.
The host cast differs from the copy kernel on GPU for NaN, inf and values out of the int64 range, so the result depends on whether the right side is 0-d. The docstring says it casts as the copy kernel would:
import numpy as np
import dpnp.tensor as dpt
a = dpt.zeros((), dtype="i8")
a[...] = np.inf
b = dpt.zeros(1, dtype="i8")
b[...] = np.array([np.inf])
print(dpt.asnumpy(a), dpt.asnumpy(b)) # GPU: -9223372036854775808 [9223372036854775807]; master: both 9223372036854775807These values are undefined in C++, so this may be fine, but then the docstring should not promise the same cast. Is it intended?
| def _raise_on_host_copy(*args, **kwargs): | ||
| raise AssertionError("synchronizing host copy was used") |
There was a problem hiding this comment.
Nit: _raise_on_host_copy and the monkeypatch around it are repeated in test_usm_ndarray_ctor.py, test_tensor_asarray.py and test_elementwise_classes.py. A fixture in dpnp/tests/tensor/helper (e.g. no_host_copy) would keep it in one place.
This PR makes binary elementwise functions with a Python or NumPy scalar operand asynchronous, closes #3148
A scalar operand (
2 * x,x == 0,x * np.float32(2)), the scalar ofclipandwhere, and a scalar assigned through basic indexing (x[0] = 1) were turned into a 0-d array by_copy_numpy_ndarray_into_usm_ndarray. That copy waits for all work already submitted to the queue, so each of these calls blocked the host until the device was idle._copy_from_numpy_intoanddpnp.tensor.asarraynow fill 0-d input with_full_usm_ndarray, ordered after the queue's submitted events, instead. Arrays withndim > 0still go through the copyThe value is cast on the host the way the copy kernel would cast it, so results don't change
float32/complex64on devices without fp64int64, so it wraps like the copy kernel where a direct NumPy cast saturates (e.g.1e10intoint16gives-7168)The new tests patch
_copy_numpy_ndarray_into_usm_ndarrayto raise, and coverasarrayof Python scalars, NumPy scalars and 0-d arrays of every dtype, casts to another dtype, fp64 scalars on devices without fp64, indexed assignment, and binary operators,out=and in-place operators with a scalar operand. They fail without the fix (131 cases). The affecteddpnp.tensortests anddpnp/tests/test_arraycreation.pypass on a Level Zero iGPU and the OpenCL CPU device. The fp64-less test only skipped here, since both devices support fp64With the reproducer from #3148 on the iGPU, the host time of one call issued right after a ~75 ms matmul:
x * x2 * xx == 0x * np.float32(2)x[0] = 1