Skip to content

Make scalar operands of binary elementwise functions asynchronous - #3182

Open
ndgrigorian wants to merge 3 commits into
masterfrom
make-scalar-operands-in-binary-ops-asynchronous
Open

ndgrigorian wants to merge 3 commits into
masterfrom
make-scalar-operands-in-binary-ops-asynchronous

Conversation

@ndgrigorian

@ndgrigorian ndgrigorian commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

This PR makes binary elementwise functions with a Python or NumPy scalar operand asynchronous, closes #3148

A scalar operand (2 * x, x == 0, x * np.float32(2)), the scalar of clip and where, and a scalar assigned through basic indexing (x[0] = 1) were turned into a 0-d array by _copy_numpy_ndarray_into_usm_ndarray. That copy waits for all work already submitted to the queue, so each of these calls blocked the host until the device was idle. _copy_from_numpy_into and dpnp.tensor.asarray now fill 0-d input with _full_usm_ndarray, ordered after the queue's submitted events, instead. Arrays with ndim > 0 still go through the copy

The value is cast on the host the way the copy kernel would cast it, so results don't change

  • Through float32/complex64 on devices without fp64
  • Complex to real keeps the real part
  • Float to integer goes through int64, so it wraps like the copy kernel where a direct NumPy cast saturates (e.g. 1e10 into int16 gives -7168)
  • No NumPy cast warnings

The new tests patch _copy_numpy_ndarray_into_usm_ndarray to raise, and cover asarray of Python scalars, NumPy scalars and 0-d arrays of every dtype, casts to another dtype, fp64 scalars on devices without fp64, indexed assignment, and binary operators, out= and in-place operators with a scalar operand. They fail without the fix (131 cases). The affected dpnp.tensor tests and dpnp/tests/test_arraycreation.py pass on a Level Zero iGPU and the OpenCL CPU device. The fp64-less test only skipped here, since both devices support fp64

With the reproducer from #3148 on the iGPU, the host time of one call issued right after a ~75 ms matmul:

call master this PR
x * x 0.45 ms 0.51 ms
2 * x 69.37 ms 1.23 ms
x == 0 80.70 ms 0.92 ms
x * np.float32(2) 75.50 ms 1.32 ms
x[0] = 1 76.07 ms 0.43 ms
  • Have you provided a meaningful PR description?
  • Have you added a test, reproducer or referred to an issue with a reproducer?
  • Have you tested your changes locally for CPU and GPU devices?
  • Have you made sure that new changes do not introduce compiler warnings?
  • Have you checked performance impact of proposed changes?
  • Have you added documentation for your changes, if necessary?
  • Have you added your changes to the changelog?

A Python or NumPy scalar was turned into a 0-d array, or assigned through
basic indexing, with _copy_numpy_ndarray_into_usm_ndarray. That copy
waits for all work submitted to the queue, so every binary operation with
a scalar operand (2 * x, x == 0), clip, where and x[0] = 1 blocked the
host until the device was idle (gh-3148).

_copy_from_numpy_into and asarray now fill 0-d input with
_full_usm_ndarray. The value is cast on the host the way the copy kernel
casts it: through float32/complex64 on devices without fp64, the real
part of complex values, float-to-integer through int64 so that it wraps,
and without NumPy's cast warnings.
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

View rendered docs @ https://intelpython.github.io/dpnp/pull/3182/index.html

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Array API standard conformance tests for dpnp=0.22.0dev1=np2py314h8d9cdd5_9 ran successfully.
Passed: 1491
Failed: 0
Skipped: 6

@coveralls

coveralls commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Coverage Status

coverage: 84.404%. remained the same — make-scalar-operands-in-binary-ops-asynchronous into master

@antonwolfy

Copy link
Copy Markdown
Contributor

@ndgrigorian, would it make sense to extend ASV scope with a benchmark on scalar operands in binary elementwise functions? I guess we not need to cover all elementwise functions, but few or couple of most valuable ones.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Replacing SYCL casts with host NumPy casts still needs device-level validation for NaN, infinity, and out-of-int64-range values, including on fp64-less devices not covered by the reported runs.

No open issues 🎉

What changed in this PR

This PR addresses #3148 by replacing synchronizing host copies of scalar operands with asynchronous device fills in dpnp’s tensor layer, allowing scalar operations to return without waiting for queued device work.

Changes:

  • Uses device fills for scalar and 0-d NumPy inputs in array creation and basic indexed assignment.
  • Adds host-side casting intended to preserve copy-kernel conversion behavior.
  • Adds regression tests for scalar creation, assignment, and binary operations, and documents the change.
File Description
dpnp/​tests/​tensor/​test_usm_ndarray_ctor.py Tests scalar assignment without host copies and checks casting behavior.
dpnp/​tests/​tensor/​test_tensor_asarray.py Tests scalar creation, dtype conversion, and devices without fp64 support.
dpnp/​tests/​tensor/​elementwise/​test_elementwise_classes.py Tests scalar binary operands, including reversed, output, and in-place operations.
dpnp/​tensor/​_ctors.py Routes 0-d NumPy inputs through device fills.
dpnp/​tensor/​_copy_utils.py Adds scalar casting and asynchronous fills with event tracking.
CHANGELOG.md Documents the removal of blocking scalar host copies.

Review effort: Balanced

@antonwolfy antonwolfy added this to the 0.22.0 release milestone Nov 9, 2026
Comment on lines +139 to +143
if np_ary.ndim == 0:
# fill instead of the host copy, which waits for the whole queue
_manager = SequentialOrderManager[dst.sycl_queue]
_fill_from_numpy_scalar(dst, np_ary, _manager.submitted_events)
return

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

x[...] = 1 now returns before the fill runs. test_ctor_buffer_kwarg (test_usm_ndarray_ctor.py:533-537) reads the memory with usm_data.copy_to_host(), which does not wait for the fill on an out-of-order queue. On the OpenCL CPU (what CI runs) the test fails in 186 of 2000 runs; on master it never fails:

import numpy as np
import dpnp.tensor as dpt

stale = 0
for _ in range(2000):
    x = dpt.usm_ndarray((10, 5), dtype="c8")
    x[...] = 1
    h = np.frombuffer(x.usm_data.copy_to_host(), dtype="c8")
    stale += not (h == 1).all()
print(f"stale in {stale}/2000 runs")  # CPU: 210/2000, GPU: 0/2000; master: 0/2000

Could the test call Xusm.sycl_queue.wait() before copy_to_host()?

Comment on lines +126 to +130
fill_val = np_ary.astype(dst_dt)[()].item()
hev, fill_ev = ti._full_usm_ndarray(
fill_val, dst, sycl_queue=copy_q, depends=depends
)
SequentialOrderManager[copy_q].add_event_pair(hev, fill_ev)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assigning a scalar to a strided view is slower on GPU now: the strided _full_usm_ndarray path packs the shape into a device allocation and adds two host tasks per call, where the old blocking copy added none:

import time, dpnp

z = dpnp.zeros((2048, 2048), dtype="f4")

def fill_columns():
    for i in range(200):
        z[:, i] = 1.0
    z.sycl_queue.wait()

fill_columns()
times = []
for _ in range(10):
    t = time.perf_counter()
    fill_columns()
    times.append((time.perf_counter() - t) * 1e3)
print(f"{sorted(times)[5]:.1f} ms")  # GPU Max 1100: 24 ms; master: 11 ms (CPU: 35 ms on both)

After other work on the queue it gets worse (96 ms vs 11 ms in a longer run). Is this expected, or could the strided fill avoid the extra host tasks?

Comment on lines +118 to +127
np_ary = np_ary.real
# the copy kernel casts NaN and out-of-range values without warning
with np.errstate(all="ignore"):
if np_ary.dtype.kind == "f" and dst_dt.kind in "iu":
# the copy kernel narrows through int64, so it wraps like NumPy
# integer casts where a direct NumPy float cast saturates
if dst_dt.itemsize < 8 or (dst_dt.kind == "u" and np_ary < 0):
np_ary = np_ary.astype(np.int64)
fill_val = np_ary.astype(dst_dt)[()].item()
hev, fill_ev = ti._full_usm_ndarray(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The host cast differs from the copy kernel on GPU for NaN, inf and values out of the int64 range, so the result depends on whether the right side is 0-d. The docstring says it casts as the copy kernel would:

import numpy as np
import dpnp.tensor as dpt

a = dpt.zeros((), dtype="i8")
a[...] = np.inf
b = dpt.zeros(1, dtype="i8")
b[...] = np.array([np.inf])
print(dpt.asnumpy(a), dpt.asnumpy(b))  # GPU: -9223372036854775808 [9223372036854775807]; master: both 9223372036854775807

These values are undefined in C++, so this may be fine, but then the docstring should not promise the same cast. Is it intended?

Comment on lines +754 to +755
def _raise_on_host_copy(*args, **kwargs):
raise AssertionError("synchronizing host copy was used")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: _raise_on_host_copy and the monkeypatch around it are repeated in test_usm_ndarray_ctor.py, test_tensor_asarray.py and test_elementwise_classes.py. A fixture in dpnp/tests/tensor/helper (e.g. no_host_copy) would keep it in one place.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python scalar operands in binary elementwise functions synchronize the queue

5 participants