llama.cpp/ggml/src
Alan Gray 3f9da22c2b
Simplify and improve CUDA graphs through use of indirect copy pointers (#9017)
* CUDA: Simplify and improve CUDA graphs through use of indirect copy pointers

Previously there was complexity in the CUDA graphs implementation due
frequently changing parameters to copy kernels associated with K and V
cache pointers. This patch simplifies by using indirection to avoid
such parameters frequently changing, avoiding the need for frequent
graph updates.

Fixes #12152

* Addressed comments

* fix HIP builds

* properly sync to stream

* removed ggml_cuda_cpy_fn_ptrs

* move stream sync before free

* guard to only use indirection with graphs

* style fixes

* check for errors

---------

Co-authored-by: slaren <slarengh@gmail.com>
2025-04-03 03:31:15 +02:00
..
ggml-blas ggml : add support for dynamic loading of backends (#10469) 2024-11-25 15:13:39 +01:00
ggml-cann CANN: Fix failed test cases (#12708) 2025-04-03 08:49:51 +08:00
ggml-cpu cpu : rm unused variable (ggml/1166) 2025-03-30 08:33:31 +03:00
ggml-cuda Simplify and improve CUDA graphs through use of indirect copy pointers (#9017) 2025-04-03 03:31:15 +02:00
ggml-hip HIP: implement FlashAttention via rocWMMA for CDNA and RDNA3+ (#12032) 2025-03-03 22:10:54 +01:00
ggml-kompute llama : add Qwen2VL support + multimodal RoPE (#10361) 2024-12-14 14:43:46 +02:00
ggml-metal metal : use F32 prec in FA kernels (#12688) 2025-04-01 14:57:19 +03:00
ggml-musa cuda : enable CUDA Graph on CUDA Toolkit < 12.x (#12394) 2025-03-17 20:25:13 +02:00
ggml-opencl opencl: use `max_alloc_size` in backend ctx instead of querying again (#12705) 2025-04-02 17:01:42 -07:00
ggml-rpc rpc : send hash when tensor data is above some fixed threshold (#12496) 2025-03-28 08:18:04 +02:00
ggml-sycl SYCL: Rename oneMKL to oneMath (#12192) 2025-04-01 16:24:29 +08:00
ggml-vulkan vulkan: Implement split_k for coopmat2 flash attention. (#12627) 2025-04-02 14:25:08 -05:00
CMakeLists.txt cmake : fix ccache conflict (#12522) 2025-03-29 11:04:58 +01:00
ggml-alloc.c ggml : upgrade init_tensor API to return a ggml_status (#11854) 2025-02-28 14:41:47 +01:00
ggml-backend-impl.h ggml : upgrade init_tensor API to return a ggml_status (#11854) 2025-02-28 14:41:47 +01:00
ggml-backend-reg.cpp ggml-backend : fix backend search path (#12330) 2025-03-11 14:25:17 +01:00
ggml-backend.cpp ggml : portability fixes for VS 2017 (#12150) 2025-03-04 18:53:26 +02:00
ggml-common.h musa: fix all warnings, re-enable `-DLLAMA_FATAL_WARNINGS=ON` in ci and update doc (#12611) 2025-03-30 10:59:38 +02:00
ggml-impl.h ggml : riscv: add 128-bit RVV support (#12530) 2025-03-27 08:38:34 +02:00
ggml-opt.cpp ggml-opt: fix data corruption (ggml/1022) 2024-11-21 09:22:02 +02:00
ggml-quants.c ggml : portability fixes for VS 2017 (#12150) 2025-03-04 18:53:26 +02:00
ggml-quants.h ggml : build backends as libraries (#10256) 2024-11-14 18:04:35 +01:00
ggml-threading.cpp ggml : build backends as libraries (#10256) 2024-11-14 18:04:35 +01:00
ggml-threading.h remove CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS (#10797) 2024-12-12 19:02:49 +01:00
ggml.c llama : add option to override model tensor buffers (#11397) 2025-04-02 14:52:01 +02:00
gguf.cpp Fix clang warning in gguf_check_reserved_keys (#12686) 2025-04-01 13:12:53 +02:00