* first commit naive test to enable mmq for RDNA4
* adding appropriate WMMA instructions
* git rebase on top of master: fixing the correctness of the mat mul operations, updating layout mappings for RDNA4
* clean up merge conflicts
* add comments and code clean up
* PR clean up, addressed comments
* enable MMQ fallback on RDNA4
* addressed comments: add guards in load generic, separate wmma branch for use_mmq function
* Revert build-xcframework.sh
* Formating: remove trailing whitespace
* revert CMake files
* clean up after rebase: remove duplicated change, revert cmake files
* clean up after rebase: revert changes from build-xcframework.sh
* clean up: remove extra space line in mma.cuh
* Revert "clean up: remove extra space line in mma.cuh"
This reverts commit b39ed57c45.
This commit removes the "-dirty" suffix from the GGML version string.
The motivation for this change is to ensure that the version string
works with different ways of checking out ggml and using it in projects.
By removing the dirty flag from the version string, we avoid potential
artifacts like shared libraries getting a -dirty suffix in their names.
Instead, if the project is built from a dirty git state, the dirty flag
will be appended to the commit hash in the GGML_BUILD_COMMIT variable.
This will enable users to still identify that the build was made from
from a modified/dirty state even though the version might match a "real"
version.
For example, the commit can be produces as follows:
```c++
printf("commit: %s\n", ggml_commit());
```
Which would print the following for a dirty build:
```console
commit: 781baf2a-dirty
```
Refs: https://github.com/ggml-org/ggml/pull/1363#issuecomment-3569691546
* ggml: add RISC-V cpu-feats
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
* fix comment[1]
---------
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
**Description of the problem**
`cann_graph_update_required` is redundantly defined and
initialized as `false` inside two mutually exclusive macro branches.
**Proposed solution**
Define it right before the macro so that it could serve both
branches.
* ggml-hexagon: fix build error with GCC
Add stdexcept include to fix GCC build errors
Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
* ggml-hexagon: check VTCM acquire failures
Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
* ggml-hexagon: disable destination bypass on older than v73
v68 errors out if having bypass enabled when the VTCM is the destination.
At least on v68 this made things actually work... not a proper fix though, so to look at later...
Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
* ggml-hexagon: add initial v68/v69 support
v68 is the Hexagon revision notably used on the Snapdragon 8cx
Gen 3 and the QCM6490.
Also add support for v69.
8MB isn't a supported page size, so relax asked for page size constraint
for HAP_compute_res_attr_set_vtcm_param_v2 to optimal.
Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---------
Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
* hexagon: add buffer support checks for hexagon sessions
* refactor: simplify buffer support checks in hexagon operations
* hexagon: update buffer support checks to use tensor structure
* refactor: streamline buffer initialization for DSP queue in hexagon operations
* refactor: simplify buffer initialization in DSP queue for hexagon operations
* refactor: optimize hex_supported_buffer function by fold expression
* wip
* refactor: simplify dspqueue_buffers_init function and its usage in hexagon operations
* fix: improve nan handling at hvx_vec_fast_sigmoid_fp32_guard
* refactor: optimize hvx_vec_inverse_fp32_guard for better nan handling
* refactor: update hvx_vec_fast_sigmoid_fp32_guard to use adjusted exponent limits
* refactor: modify hvx_vec_fast_sigmoid_fp32_guard to accept parameters for improved flexibility
* refactor: update hvx_vec_exp_fp32_guard to accept max_exp and inf parameters to save some instructions
* refactor: move hvx_vec_inverse_fp32_guard implementation to hvx-inverse.c for better perf
* mmf for rdna4
* align the padding for rdna4
* forbit mul_mat_f for rdna4
* fix as comment
* remove device kernels
* add constexpr for early return
* update based on review comment
* change based on the review comment
* pass compile error
* keep code consistency
---------
Co-authored-by: zhang hui <you@example.com>
* CANN: Refactor `evaluate_and_capture_cann_graph`
**Description of the problem**
* `matched_graph` is obtained even if graph mode is disabled.
* End of graph capture and graph replay are unnecessarily placed in different `if` blocks.
**Proposed solution**
* Obtain `matched_graph` only if graph mode is enabled.
* Place end of graph capture and graph reply inside the same `if` block.
* Unify graph related comments.
* Remove trailing whitespace
* vulkan: support larger argsort
This is an extension of the original bitonic sorting shader that puts the
temporary values in global memory and when more than 1024 threads are needed
it runs multiple workgroups and synchronizes through a pipelinebarrier.
To improve the memory access pattern, a copy of the float value is kept with
the index value. I've applied this same change to the original shared memory
version of the shader, which is still used when ncols <= 1024.
* Reduce the number of shader variants. Use smaller workgroups when doing a single pass, for a modest perf boost
* reduce loop overhead
* run multiple cols per invocation, to reduce barrier overhead
* Fix too relaxed check on CUDA "fast copy" (can_be_transposed) condition
* Argh.
* Making CISC happy ;)
* Integrate CONT tests
* Use loopy loop
* Skip new tests for (B)F16 for now.
* cann: fix acl_tensor_ptr usage in ASCEND_310P ROPE implementation
Fix compilation errors in the ASCEND_310P-specific ROPE operation code
by adding .get() calls when passing acl_tensor_ptr smart pointers to
functions expecting raw aclTensor* pointers.
This fixes the code that was missed in the previous refactoring commit
(8981848) which changed ggml_cann_create_tensor() return type from
aclTensor* to acl_tensor_ptr.
* cann: format code
* CANN: Use smart pointers to manage ACL objects
Previously, ACL objects were managed via manual destruction, which
led to multiple memory-leak issues during runtime. This patch replaces
manual memory management with smart pointers so that ACL objects
are properly released and ownership is clearly defined.
Note that the ownership of an ACL object belongs to the function
that creates it. Other internal functions should operate on these ACL
objects using raw pointers to avoid unintended ownership transfers.
Additionally, since aclTensorList automatically frees its contained
aclTensor objects, any aclTensor added to a tensor list must release
ownership to avoid double free operations.
This PR also removes the asynchronous task submission mechanism.
Due to changes in recent CANN versions, tiling time has significantly
decreased. Even with a dual-thread submission model, the dispatch
overhead still falls on the critical path, making async submission
less beneficial. Moreover, aclGraph support provides a much better
path to reducing operator dispatch latency.
* CANN: resolve review comments
* vulkan: add LOG operation support for F32 and F16
Part of #14909.
* vulkan: Fix LOG operation types
* docs: Update operation support documentation for Vulkan LOG operation
* vulkan: fix log_f16 shader
* docs: restore missing LOG test cases and regenerate ops.md
* SYCL: add generic unary op implementation for multiple ops (ABS/SGN/…); unify non-contiguous access
* SYCL: update documentation and sycl.csv to reflect new unary op support
* update ops.md after syncing SYCL.csv changes
* Fix SYCL.csv merge conflict
* Update ops.md after fixing SYCL.csv conflicts
* Fix SYCL.csv tail after merge conflict and regenerate ops.md
* Fix line endings and final newline in SYCL.csv
* Remove TOPK_MOE entries from SYCL.csv as requested
* Update ops.md after removing TOPK_MOE from SYCL.csv
* Regenerated SYCL.csv and synced ops.md with upstream
* Update ops.md using create_ops_docs.py
* vulkan: change graph_compute to be async and enable get_tensor_async
This allows some additional CPU/GPU overlap for large pp workloads. Also seems
to help a bit for token gen, maybe getting rid of a small bubble between
graph_compute and get_tensor.
Async set and copy functions seem to be very rarely used, so I didn't enable
them because I didn't have a good way to test them.
The async commands need to be ordered against each other, so put them all on
the compute queue. The non-async commands still use the transfer queue.
The fence for graph_compute/get_tensor_async is submitted and waited on in
ggml_vk_synchronize.
* fix thread safety errors
* teardown context cleanly
* Handle async read to non-pinned dst
* ggml-cpu: handle 3d tensors in repack mul_mat
* Removed unnecessary branch, removed need for <algorithm>
* Fixed dst_ptr pointer in chunk + clang_format
* GGML_ASSERT to check wdata within bounds
* Accidental ggml.h inclusion
* Improved GGML_ASSERT on wdata boundaries
* Address performance regression in Qwen and llama.cpp due to chunking
* vulkan: remove shell call from vulkan-shaders-gen tool
* use string vector for command execution
* Fix condition
* use string, remove const_cast
* Fix dependency file quotation on Windows
---------
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* update L2_NORM op support
* update L2_NORM op support
* remove extra whitespace
* cann: update cross_entropy_loss op support
* remove trailing whitespaces
* rebase the latest code in the main repository and remove the l2_norm operator that already exists in another pull request.
* undo the l2_norm operator deletion
* CUDA: add fused rope
* move k forward_expand up
* create helper function instead of re-using params
* make assert statement more in line with comment
* rope_norm: coalesced writes to global mem
* hexagon: explicitly check for ops with zero nrows
llm_graph_context::build_inp_out_ids() can generate tensors with zero nrows.
Somehow other backends seems to handle this without obvious explicit checks.
In the hexagon case we need to check explicitly and skip them.
* hexagon: introduce fastdiv, fix test-backend-ops for ADD/SUB/MUL
Co-authored-by: chraac <chraac@gmail.com>
* hexagon: use fastdiv in ADD_ID
* hexagon: use ggml_op_is_empty and ggml_is_empty to check for NOPs
---------
Co-authored-by: chraac <chraac@gmail.com>
* extract rotate_pairs logic from ggml_compute_forward_rope_f32
* templateify ggml_compute_forward_rope_f32 and _f16
* abort when rope type not supported, remove GLM from test-rope
* add imrope branch to switch
* add rope tests for perf
* Update ggml/src/ggml-cpu/ops.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Update ggml/src/ggml-cpu/ops.cpp
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
When compiling llama.cpp in Yocto, it fails QA checks because the generated so files aren't versioned. This applies a version to all generated so files, allowing the package to build without errors.
* add i8mm route with SVE ggml_vec_dot_q4_K_q8_K and ggml_vec_dot_q6_K_q8_K
* Surround SVE function with compiler directive
* fix compile switch
* fix coding style
* ggml : fix indent
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* vulkan : implement upscale with bicubic interpolation
* cuda : implement upscale with bicubic interpolation
* tests : add ggml_interpolate with GGML_SCALE_MODE_BICUBIC to backend tests
* adapt OpenCL backend to not support the OP in that case so tests don't fail
* print scale mode & flags in test-backend-ops
* vulkan: use all device-local heaps for memory availability reporting
Co-authored-by: Giuseppe Scrivano <gscrivan@redhat.com>
* use all available heaps for iGPU memory reporting
* Allow multiple memory types per buffer request for devices with split heaps
---------
Co-authored-by: Giuseppe Scrivano <gscrivan@redhat.com>
This change combines the rms_norm+mul and rope+view+set_rows fusions to
allow fusing the whole sequence together. This comes up in Qwen3, Bailing,
and some other models.
The std::map pipeline_flash_attn_f32_f16 could be searched and inserted at the
same time, which needs to hold the lock. To be safe, hold the lock for all of
ggml_vk_load_shaders.
Jeff Bolz
Jiacheng (Jason) Chen
Daniel Bevenius
Alberto Cabrera Pérez
ixgbe
Max Krasnyansky
Raul Torres
M. Mediouni
nullname
Sigbjørn Skjæret
yulo
lhez
YangLe
Piotr Wilkin (ilintar)
Scott Fudally
Adrien Gallouët
sudhiarm
Giuseppe Scrivano
Aman Gupta
Ruben Ortlam
Jeremy Rand
Chenguang Li
Georgi Gerganov
hipudding
Pavels Zaicenkovs
shaofeiqi
shani-f
Alberto Cabrera Pérez
Diego Devesa
bagheera
TecJesh
Johannes Gäßler
Neo Zhang Jianyu
Eve
duduta
Charles Xu
Mike Abbott
fj-y-saito
Acly
SavicStefan
Aleksei Nikiforov
Reese Levine
bssrdf