llama.cpp/ggml/src
Jeff Bolz 611f419cff
vulkan: optimize rms_norm, and allow the work to spread across multiple SMs (#15281)
* vulkan: optimize rms_norm, and allow the work to spread across multiple SMs

There are really two parts to this change:
(1) Some optimizations similar to what we have in soft_max, to unroll with
different numbers of iterations.
(2) A fusion optimization where we detect add followed by rms_norm, and make
the add shader atomically accumulate the values^2 into memory. Then the
rms_norm shader can just load that sum. This allows the rms_norm to be
parallelized across multiple workgroups, it just becomes a simple per-element
multiply.

The fusion optimization is currently only applied when the rms_norm is on a
single vector. This previously always ran on a single SM. It could apply more
broadly, but when there are other dimensions the work can already spread across
SMs, and there would be some complexity to tracking multiple atomic sums.

* Change add+rms_norm optimization to write out an array of partial sums
rather than using atomic add, to make it deterministic. The rms_norm
shader fetches a subgroup's worth in parallel and uses subgroupAdd to
add them up.

* complete rebase against fused adds - multi_add shader can also compute partial sums

* fix validation errors

* disable add_rms_fusion for Intel due to possible driver bug

* resolve against #15489, sync after clearing partial sums
2025-08-23 13:16:17 -05:00
..
ggml-blas ggml : fix field name when new ggml_backend (#14944) 2025-08-08 14:37:22 +02:00
ggml-cann CANN: Optimize RMS_NORM using cache (#15419) 2025-08-22 14:12:07 +08:00
ggml-cpu ggml: add `conv3d` op (#15182) 2025-08-22 15:33:15 +02:00
ggml-cuda vulkan : support ggml_mean (#15393) 2025-08-23 08:35:21 +02:00
ggml-hip HIP: bump requirement to rocm 6.1 (#15296) 2025-08-13 20:44:30 +02:00
ggml-metal ggml : fix condition of im2col on Metal backend (#15460) 2025-08-21 08:32:26 +03:00
ggml-musa CUDA: replace GGML_CUDA_F16 with CUDA arch checks (#15433) 2025-08-20 16:58:49 +02:00
ggml-opencl opencl: mark `argsort` unsupported if cols exceed workgroup limit (#15375) 2025-08-19 11:25:51 -07:00
ggml-rpc ggml-rpc: chunk send()/recv() to avoid EINVAL for very large tensors over RPC (macOS & others) (#15188) 2025-08-13 08:54:30 +03:00
ggml-sycl vulkan : support ggml_mean (#15393) 2025-08-23 08:35:21 +02:00
ggml-vulkan vulkan: optimize rms_norm, and allow the work to spread across multiple SMs (#15281) 2025-08-23 13:16:17 -05:00
ggml-webgpu ggml WebGPU: add support for quantization types (#15440) 2025-08-22 11:28:03 -07:00
ggml-zdnn ggml: initial IBM zDNN backend (#14975) 2025-08-15 21:11:22 +08:00
CMakeLists.txt ggml: initial IBM zDNN backend (#14975) 2025-08-15 21:11:22 +08:00
ggml-alloc.c llama : add gpt-oss (#15091) 2025-08-05 22:10:36 +03:00
ggml-backend-impl.h ggml : upgrade init_tensor API to return a ggml_status (#11854) 2025-02-28 14:41:47 +01:00
ggml-backend-reg.cpp ggml: initial IBM zDNN backend (#14975) 2025-08-15 21:11:22 +08:00
ggml-backend.cpp sched : fix possible use of wrong ids tensor when offloading moe prompt processing (#15488) 2025-08-21 23:09:32 +02:00
ggml-common.h llama : add gpt-oss (#15091) 2025-08-05 22:10:36 +03:00
ggml-impl.h llama : add gpt-oss (#15091) 2025-08-05 22:10:36 +03:00
ggml-opt.cpp finetune: SGD optimizer, more CLI args (#13873) 2025-08-14 12:03:57 +02:00
ggml-quants.c ggml-quants : fix make_qp_quants NANs and IQ1 assertion errors (#15379) 2025-08-18 09:23:56 +02:00
ggml-quants.h llama : add gpt-oss (#15091) 2025-08-05 22:10:36 +03:00
ggml-threading.cpp ggml : build backends as libraries (#10256) 2024-11-14 18:04:35 +01:00
ggml-threading.h remove CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS (#10797) 2024-12-12 19:02:49 +01:00
ggml.c ggml: add `conv3d` op (#15182) 2025-08-22 15:33:15 +02:00
ggml.cpp ggml : Print backtrace on uncaught C++ exceptions (ggml/1232) 2025-06-01 13:43:57 +03:00
gguf.cpp ggml : prevent integer overflow in gguf tensor size calculation (#14595) 2025-07-09 14:33:53 +02:00