happyz
/
llama.cpp
mirror of https://github.com/ggerganov/llama.cpp.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
llama.cpp/ggml/src/ggml-cuda/template-instances
History
Aman Gupta a972faebed
CUDA: Add mul_mat_id support for the mmf kernel (#15767) 
* CUDA: Add mul_mat_id support the mmf

Add support for mul_mat_id for bs < 16

* Review: use warp_size, fix should_use_mmf condition

* Launch one block per expert, stride along n_expert_used

* templatize mul_mat_id

* Pad shmem to 16 bytes, add helper function mul_mat_f_switch_ids

* Reduce compile times by dividing mmf into f16, bf16 and f32 variants

* Divide mmf by ncols_dst

* Add missing files

* Fix MUSA/HIP builds
 		2025-09-09 14:38:02 +08:00
..
fattn-mma-f16-instance-ncols1_1-ncols2_8.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_1-ncols2_16.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_2-ncols2_4.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_2-ncols2_8.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_2-ncols2_16.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_4-ncols2_2.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_4-ncols2_4.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_4-ncols2_8.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_4-ncols2_16.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_8-ncols2_1.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_8-ncols2_2.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_8-ncols2_4.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_8-ncols2_8.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_16-ncols2_1.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_16-ncols2_2.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_16-ncols2_4.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_32-ncols2_1.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_32-ncols2_2.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-mma-f16-instance-ncols1_64-ncols2_1.cu CUDA: FA support for Deepseek (Ampere or newer) (#13306) 2025-05-09 13:34:58 +02:00
fattn-vec-f16-instance-hs64-f16-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs64-f16-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs64-f16-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs64-f16-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs64-f16-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs64-f16-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-f16-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-f16-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-f16-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-f16-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-f16-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-f16-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q4_0-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q4_0-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q4_0-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q4_0-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q4_0-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q4_0-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q4_1-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q4_1-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q4_1-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q4_1-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q4_1-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q4_1-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q5_0-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q5_0-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q5_0-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q5_0-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q5_0-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q5_0-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q5_1-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q5_1-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q5_1-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q5_1-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q5_1-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q5_1-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q8_0-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q8_0-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q8_0-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q8_0-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q8_0-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs128-q8_0-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f16-instance-hs256-f16-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs64-f16-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs64-f16-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs64-f16-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs64-f16-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs64-f16-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs64-f16-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-f16-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-f16-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-f16-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-f16-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-f16-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-f16-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q4_0-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q4_0-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q4_0-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q4_0-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q4_0-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q4_0-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q4_1-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q4_1-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q4_1-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q4_1-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q4_1-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q4_1-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q5_0-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q5_0-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q5_0-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q5_0-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q5_0-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q5_0-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q5_1-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q5_1-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q5_1-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q5_1-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q5_1-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q5_1-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q8_0-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q8_0-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q8_0-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q8_0-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q8_0-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs128-q8_0-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
fattn-vec-f32-instance-hs256-f16-f16.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
generate_cu_files.py CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_1.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_2.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_3.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_4.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_5.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_6.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_7.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_8.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_9.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_10.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_11.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_12.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_13.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_14.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_15.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmf-instance-ncols_16.cu CUDA: Add mul_mat_id support for the mmf kernel (#15767) 2025-09-09 14:38:02 +08:00
mmq-instance-iq1_s.cu CUDA: MMQ code deduplication + iquant support (#8495) 2024-07-20 22:25:26 +02:00
mmq-instance-iq2_s.cu CUDA: MMQ code deduplication + iquant support (#8495) 2024-07-20 22:25:26 +02:00
mmq-instance-iq2_xs.cu CUDA: MMQ code deduplication + iquant support (#8495) 2024-07-20 22:25:26 +02:00
mmq-instance-iq2_xxs.cu CUDA: MMQ code deduplication + iquant support (#8495) 2024-07-20 22:25:26 +02:00
mmq-instance-iq3_s.cu CUDA: MMQ code deduplication + iquant support (#8495) 2024-07-20 22:25:26 +02:00
mmq-instance-iq3_xxs.cu CUDA: MMQ code deduplication + iquant support (#8495) 2024-07-20 22:25:26 +02:00
mmq-instance-iq4_nl.cu CUDA: MMQ support for iq4_nl, iq4_xs (#8278) 2024-07-05 09:06:31 +02:00
mmq-instance-iq4_xs.cu CUDA: MMQ support for iq4_nl, iq4_xs (#8278) 2024-07-05 09:06:31 +02:00
mmq-instance-mxfp4.cu llama : add gpt-oss (#15091) 2025-08-05 22:10:36 +03:00
mmq-instance-q2_k.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
mmq-instance-q3_k.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
mmq-instance-q4_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
mmq-instance-q4_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
mmq-instance-q4_k.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
mmq-instance-q5_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
mmq-instance-q5_1.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
mmq-instance-q5_k.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
mmq-instance-q6_k.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00
mmq-instance-q8_0.cu llama : reorganize source code + improve CMake (#8006) 2024-06-26 18:33:02 +03:00