Skip to content

fix: Pass fp_mixed_precision to the reshaped kernel in ClGemm - #1325

Open
nirvanahyj47 wants to merge 2 commits into
ARM-software:mainfrom
nirvanahyj-com:fix-clgemm-fp-mixed-precision
Open

nirvanahyj47 wants to merge 2 commits into
ARM-software:mainfrom
nirvanahyj-com:fix-clgemm-fp-mixed-precision

Conversation

@nirvanahyj47

Copy link
Copy Markdown

GEMMInfo::fp_mixed_precision() is never copied into the GEMMKernelInfo that ClGemm builds, so ClGemmMatrixMultiplyReshapedKernel, which adds -DMIXED_PRECISION (fp32 accumulators) when the flag is set, never sees it. An F16 CLGEMM configured with fp_mixed_precision = true gives results and speed identical to the default (max. relative error 1.95e-2 at N = 256, 4.15e-2 at N = 1024 on a Mali-T628 under Mesa/Rusticl).

Copy the flag in configure_reshaped() and validate_reshaped(), the path whose kernel implements mixed precision. The native and reshaped_only_rhs kernels reject the flag, so they keep ignoring it.

With the change, the same F16 GEMM with fp_mixed_precision = true gives a max. relative error of 3.8e-4 to 4.4e-4 for N = 64 to 1024 (no element off by more than the tolerance), at the same speed within noise (3.0 / 4.65 / 6.33 GFLOPS for N = 256 / 512 / 1024).

Found and written with Claude Code (Anthropic), a coding assistant; tested on v53.3.1 (Samsung ARTIK 10, Exynos 5422).

GEMMInfo::fp_mixed_precision() is never copied into the GEMMKernelInfo
that ClGemm builds, so ClGemmMatrixMultiplyReshapedKernel, which adds
-DMIXED_PRECISION (fp32 accumulators) when the flag is set, never sees
it. An F16 CLGEMM configured with fp_mixed_precision = true gives results
and speed identical to the default (max. relative error 1.95e-2 at
N = 256, 4.15e-2 at N = 1024 on a Mali-T628 under Mesa/Rusticl).

Copy the flag in configure_reshaped() and validate_reshaped(), the path
whose kernel implements mixed precision. The native and
reshaped_only_rhs kernels reject the flag, so they keep ignoring it.

With the change, the same F16 GEMM with fp_mixed_precision = true gives
a max. relative error of 3.8e-4 to 4.4e-4 for N = 64 to 1024 (no
element off by more than the tolerance), at the same speed within
noise (3.0 / 4.65 / 6.33 GFLOPS for N = 256 / 512 / 1024).

Found and written with Claude Code (Anthropic), a coding assistant;
tested on v53.3.1 (Samsung ARTIK 10, Exynos 5422).

Signed-off-by: Young-Jun Hong <nirvanah@nirvanahyj.com>

@morgolock morgolock left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the FP32 compatibility regression before merging.

With FP32 inputs and fp_mixed_precision=true, the new assignment makes RESHAPED fail validation with “Mixed precision only supported for F16 data type.” The same call worked before this change and still works through NATIVE and RESHAPED_ONLY_RHS.

Please gate the flag on F16 in both configure_reshaped() and validate_reshaped():

kernel_info.fp_mixed_precision =
    data_type == DataType::F16 && gemm_info.fp_mixed_precision();

I reproduced this with M=7, N=5, K=1024 and constant B on Firefly, forcing the RESHAPED dispatch. The intended FP16 accuracy improvement passed both precision-sensitive cases I tested.

Please also add a public CLGEMM regression covering FP16 mixed-precision execution and the existing FP32 behavior with the flag enabled. Existing kernel-level mixed-precision tests bypass the flag propagation being fixed here.

@morgolock

Copy link
Copy Markdown
Contributor

Could you share more about your use case for ACL?

  • What application, models, or workloads are you running?
  • Are you using ACL directly or through a framework?
  • Which hardware do you target, and do you require CPU acceleration, GPU acceleration, or both?
  • What functionality or performance requirements would a replacement need to preserve?

As ACL approaches end of life, we want to help users identify suitable migration paths, including KleidiAI or LiteRT where appropriate. Understanding your requirements will help us assess which options could work for you.

The previous change copies GEMMInfo::fp_mixed_precision() into the
GEMMKernelInfo of the reshaped kernel for every data type. An F32
CLGEMM with the flag set then fails validation of
ClGemmMatrixMultiplyReshapedKernel ("Mixed precision only supported for
F16 data type") when the reshaped kernel is selected, while the flag
used to be ignored for F32.

Copy the flag only for F16, in configure_reshaped() and
validate_reshaped().

Add CLGEMM validation tests for the flag. A new fixture sets
fp_mixed_precision, can make B and C constant (CLGEMM selects the
reshaped kernels only for a constant B), checks that CLGEMM::validate()
accepts the configuration and compares F16 results against the mixed
precision reference. The cases are F16 on the small GEMM shapes and
F32 on the small GEMM shapes plus M = 7, N = 5, K = 1024, each with a
constant and a non-constant B.

On a Mali-T628 (Mesa Rusticl), a run of the new suite fails the four
F32 cases that have a constant B and M > 1, including M = 7, N = 5,
K = 1024, at the validate() check with the previous change, and passes
all 26 cases with this change. A run of all CL GEMM precommit tests
gives the same results with and without this change; the only failures
are 14 OUTPUT_3D, INPUT_OUTPUT_3D and BATCHED_MATMUL cases that this
driver rejects at enqueue (no non-uniform work-group support).

Found and written with Claude Code (Anthropic), a coding assistant;
tested on a Samsung ARTIK 10 (Exynos 5422).

Signed-off-by: Young-Jun Hong <nirvanah@nirvanahyj.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants