llama.cpp

Commit Graph

Author	SHA1	Message	Date
Johannes Gäßler	1f0dabda8d	CUDA: use tensor cores for MMQ (#7676 ) * CUDA: int8 tensor cores for MMQ (legacy quants) * fix out-of-bounds writes * __builtin_assume -> GGML_CUDA_ASSUME * fix writeback returning too early	2024-06-10 11:45:13 +02:00
Johannes Gäßler	750f60c03e	CUDA: fix Pascal FA, deq. KV to FP16 for batch > 8 (#7681 )	2024-06-01 15:47:04 +02:00
Johannes Gäßler	9b596417af	CUDA: quantized KV support for FA vec (#7527 ) * CUDA: quantized KV support for FA vec * try CI fix * fix commented-out kernel variants * add q8_0 q4_0 tests * fix nwarps > batch size * split fattn compile via extern templates * fix flake8 * fix metal tests * fix cmake * make generate_cu_files.py executable * add autogenerated .cu files * fix AMD * error if type_v != FP16 and not flash_attn * remove obsolete code	2024-06-01 08:44:14 +02:00

Author

SHA1

Message

Date

Johannes Gäßler

1f0dabda8d

CUDA: use tensor cores for MMQ (#7676 )

* CUDA: int8 tensor cores for MMQ (legacy quants)

* fix out-of-bounds writes

* __builtin_assume -> GGML_CUDA_ASSUME

* fix writeback returning too early

2024-06-10 11:45:13 +02:00

Johannes Gäßler

750f60c03e

CUDA: fix Pascal FA, deq. KV to FP16 for batch > 8 (#7681 )

2024-06-01 15:47:04 +02:00

Johannes Gäßler

9b596417af

CUDA: quantized KV support for FA vec (#7527 )

* CUDA: quantized KV support for FA vec

* try CI fix

* fix commented-out kernel variants

* add q8_0 q4_0 tests

* fix nwarps > batch size

* split fattn compile via extern templates

* fix flake8

* fix metal tests

* fix cmake

* make generate_cu_files.py executable

* add autogenerated .cu files

* fix AMD

* error if type_v != FP16 and not flash_attn

* remove obsolete code

2024-06-01 08:44:14 +02:00

3 Commits