bd90fc74c3
ggml-webgpu: improve flastAttention performance by software pipelining ( #19151 )
...
* webgpu : pipeline flash_attn Q/K loads in WGSL
* ggml-webgpu: unroll Q*K accumlation inner loop
* ggml-webgpu: vectorization
* ggml-webgpu: unrolling
* ggml-webgpu: remove redundant unrolling
* ggml-webgpu: restore the config
* ggml-webgpu: remove redundant comments
* ggml-webgpu: formatting
* ggml-webgpu: formatting and remove vectorization
* ggml-webgpu: remove unnecessary constants
* ggml-webgpu: change QKV buffer to read_write to pass validation
* ggml-webgpu: add explanation for the additional bracket around Q K accumulate
* Indentation and for -> if for tail
* Kick off CI on wgsl only commits
---------
Co-authored-by: Reese Levine <reeselevine1@gmail.com>
2026-01-29 14:05:30 -08:00
ce38a4db47
hexagon: enable offloading to Hexagon on Windows on Snapdragon ( #19150 )
...
* hexagon: updates to enable offloading to HTP on WoS
* Update windows.md
* Update windows.md
* hexagon: enable -O3 optimizations
* hexagon: move all _WINDOWS conditional compilation to _WIN32
* hexagon: updates to enable offloading to HTP on WoS
* hexagon: use run-time vs load-time dynamic linking for cdsp driver interface
* refactor htp-drv
* hexagon: add run-bench.ps1 script
* hexagon: htdrv refactor
* hexagon: unify Android and Windows build readmes
* hexagon: update README.md
* hexagon: refactor htpdrv
* hexagon: drv refactor
* hexagon: more drv refactor
* hexagon: fixes for android builds
* hexagon: factor out dl into ggml-backend-dl
* hexagon: add run-tool.ps1 script
* hexagon: merge htp-utils in htp-drv and remove unused code
* wos: no need for getopt_custom.h
* wos: add missing CR in htpdrv
* hexagon: ndev enforecement applies only to the Android devices
* hexagon: add support for generating and signing .cat file
* hexagon: add .inf file
* hexagon: working auto-signing and improved windows builds
* hexagon: futher improve skel build
* hexagon: add rough WoS guide
* hexagon: updated windows guide
* hexagon: improve cmake handling of certs and logging
* hexagon: improve windows setup/build doc
* hexagon: more windows readme updates
* hexagon: windows readme updates
* hexagon: windows readme updates
* hexagon: windows readme updates
* hexagon: windows readme updates
* Update windows.md
* Update windows.md
* snapdragon: rename docs/backend/hexagon to docs/backends/snapdragon
Also added a power shell script to simplify build env setup.
* hexagon: remove trailing whitespace and move cmake requirement to user-presets
* hexagon: fix CMakeUserPresets path in workflow yaml
* hexagon: introduce local version of libdl.h
* hexagon: fix src1 reuse logic
gpt-oss needs a bigger lookahead window.
The check for src[1] itself being quantized was wrong.
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-01-29 12:33:21 -08:00
4fdbc1e4db
cuda : fix nkvo, offload and cuda graph node properties matching ( #19165 )
...
* cuda : fix nkvo
* cont : more robust cuda graph node property matching
* cont : restore pre-leafs implementation
* cont : comments + static_assert
2026-01-29 18:45:30 +02:00
7b7ae857f6
chat : add parsing for solar-open-100b ( #18540 )
...
* chat : add parsing for solar-open-100b
* add comments to rules
* cont : make assistant start optional
* cont : remove assistant start prefix altogether
---------
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
2026-01-29 16:06:15 +01:00
84b0a98319
webui: Update Svelte to fix effect_update_depth_exceeded errors ( #19144 )
...
The upstream fix is first available in 5.38.2, so constrain to at least
that version.
Rebuild pre-compiled webui index.html.gz based on these changes.
See also:
https://github.com/ggml-org/llama.cpp/issues/16347
https://github.com/huntabyte/bits-ui/issues/1687
https://github.com/sveltejs/svelte/issues/16548
2026-01-29 15:56:39 +01:00
b45ef2702c
jinja : do not pass empty tools and add some none filters ( #19176 )
2026-01-29 14:06:54 +01:00
f3dd7b8e68
HIP: add mmf for CDNA ( #18896 )
...
* refactor mmf rows_per_block
* speed up compile
* pass cdna compile
* fix cuda error
* clean up mmf
* f32 mmf
* clean float mma
* fix mmf error
* faster mmf
* extend tile k
* fix compile error
* Revert "extend tile k"
This reverts commit 4d2ef3d483
2026-01-29 11:10:53 +01:00
eed25bc6b0
arg : add -kvu to llama-batched-bench ( #19172 )
2026-01-29 08:50:47 +02:00
b33df266d0
ggml-zendnn : resolve ZenDNN backend cross-module symbol dependency ( #19159 )
2026-01-29 12:28:57 +08:00
3bcc990997
CUDA: refactor topk-moe to enable more models (GLM 4.7, Nemotron etc.) ( #19126 )
2026-01-29 10:31:28 +08:00
d4964a7c66
sycl: fix norm kernels: l2_norm, group_norm, rms_norm by remove assert to support more cases ( #19154 )
...
Co-authored-by: Neo Zhang Jianyu <jianyu.zhang@intel.com>
2026-01-29 09:20:22 +08:00
50e8962f79
ci : find latest release with asset for winget ( #19161 )
2026-01-28 22:05:39 +01:00
f6b533d898
Vulkan Flash Attention Coopmat1 Refactor ( #19075 )
...
* vulkan: use coopmat for flash attention p*v matrix multiplication
* fix P loading issue
* fix barrier position
* remove reduction that is no longer needed
* move max thread reduction into loop
* remove osh padding
* add bounds checks and padding
* remove unused code
* fix shmem sizes, loop duration and accesses
* don't overwrite Qf, add new shared psh buffer instead
* add missing bounds checks
* use subgroup reductions
* optimize
* move bounds check, reduce barriers
* support other Bc values and other subgroup sizes
* remove D_split
* replace Of register array with shared memory Ofsh array
* parallelize HSV across the rowgroups
* go back to Of in registers, not shmem
* vectorize sfsh
* don't store entire K tile in shmem
* fixes
* load large k tiles to shmem on Nvidia
* adapt shared memory host check function to shader changes
* remove Bc 32 case
* remove unused variable
* fix missing mask reduction tmspsh barrier
* fix mask bounds check
* fix rowmax f16 under/overflow to inf
* fix flash_attn_cm2 BLOCK_SIZE preprocessor directives
2026-01-28 18:52:45 +01:00
72d3b1898a
spec : add self‑speculative decoding (no draft model required) + refactor ( #18471 )
...
* server: introduce self-speculative decoding
* server: moved self-call into speculative.cpp
* can_speculate() includes self-speculation
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* server: can_speculate() tests self-spec
* server: replace can_speculate() with slot.can_speculate()
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* common: use %zu format specifier for size_t in logging
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* server: can_speculate() requires a task instance
* common: ngram map, config self-speculative decoding
* common: add enum common_speculative_type
* common: add vector of speculative states
* common: add option --spec-draftless
* server: cleanup (remove slot.batch_spec, rename)
* common: moved self-spec impl to ngram-map
* common: cleanup (use common_speculative_state_draft)
* spec : refactor
* cont : naming
* spec: remove --spec-config
* doc: (draftless) speculative decoding
* common: print performance in spec decoding
* minor : cleanup
* common : better names
* minor : cleanup + fix build
* minor: comments
* CODEOWNERS: add common/ngram-map.* (#18471 )
* common : rename speculative.draftless_type -> speculative.type
* ngram-map : fix uninitialized values
* ngram-map : take into account the input can become shorter
* ngram-map : revert len check for now
* arg : change `--spec-draftless` -> `--spec-type`
* spec : add common_speculative_state::accept()
* spec : refactor + add common_speculative_begin()
* spec : fix begin() call with mtmd
* spec : additional refactor + remove common_speculative_params
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
2026-01-28 19:42:42 +02:00
ebf5725870
convert : yield Mamba2Model/GraniteMoeModel modify_tensors ( #19157 )
...
* convert : yield Mamba2Model/GraniteMoeModel modify_tensors
This commit updates the `GraniteHybridModel` class' modify_tensors
function to properly delegate to `Mamba2Model.modify_tensors` and
`GraniteMoeModel.modify_tensors` using 'yield from' instead of 'return'.
The motivation for this is that modify_tensors is a generator function
(it uses 'yield from'), but the two calls above use return statements
but don't yield anything which means that the the caller of this
function will not receive any yielded values from it. And this causes
layer tensors to be silently dropped during conversion.
2026-01-28 16:49:36 +01:00
0cd7032ca4
ggml-sycl: remove unused syclcompat header ( #19140 )
...
The syclcompat/math.hpp is not used anymore. The change that intrduced it was successfuly reverted (https://github.com/ggml-org/llama.cpp/pull/17826 ).
This include path will become obsolete and dropped in oneAPI 2026.0 effectively breaking ggml-sycl builds.
2026-01-28 23:33:54 +08:00
60368e1d73
jinja : undefined should be treated as sequence/iterable (return string/array) by filters/tests ( #19147 )
...
* undefined is treated as iterable (string/array) by filters
`tojson` is not a supported `undefined` filter
* add tests
* add sequence and iterable tests
keep it DRY and fix some types
2026-01-28 14:40:29 +01:00
88d23ad515
vulkan: handle device dedup on MacOS + Vega II Duo cards ( #19058 )
...
Deduplication here relied on the fact that vulkan would return unique
UUID for different physical GPUs. It is at the moment not always the case.
On Mac Pro 2019 running Mac OS, with 2 Vega II Duo cards (so, 4 GPU total),
MotlenVK would assign same UUID to pairs of GPUs, unless they
are connected with Infinity Fabric.
See more details here: KhronosGroup/MoltenVK#2683 .
The right way is to fix that in MoltenVK, but until it is fixed,
llama.cpp would only recognize 2 of 4 GPUs in such configuration.
The deduplication logic here is changed to only filter GPUs if UUID is
same but driver is different.
2026-01-28 12:35:54 +01:00
0a95026da9
doc: add build instruction to use Vulkan backend on macos ( #19029 )
2026-01-28 12:30:16 +01:00
b7feacf7f3
ggml: new backend for Virglrenderer API Remoting acceleration (v2) ( #18718 )
2026-01-28 17:49:40 +08:00
6ad70c5a77
ggml-cpu: arm64: Q4_K scale unroll and vectorization ( #19108 )
2026-01-28 09:15:56 +02:00
631cbfcc7a
cuda : fix "V is K view" check for non-unified KV cache ( #19145 )
2026-01-28 09:15:27 +02:00
2eee6c866c
CUDA: tune GLM 4.7 Flash FA kernel selection logic (DGX Spark) ( #19142 )
2026-01-28 09:15:11 +02:00
b931f81b5a
server : adjust spec tests to generate up to 16 tokens ( #19093 )
2026-01-28 09:11:40 +02:00
c5c64f72ac
llama : disable Direct IO by default ( #19109 )
...
* llama : disable Direct IO by default
* cont : override mmap if supported
2026-01-28 09:11:13 +02:00
eef375ce16
sampling : remove sampling branching in output_reserve ( #18811 )
...
* sampling : remove sampling branching in output_reserve
This commit updates output_reserve in llama-context.cpp to always
allocate sampling buffers regardless of whether sampling is needed for
the current batch.
The motivation for this is to avoid reallocations and branching based on
the sampling requirements of the batch.
2026-01-28 05:59:30 +01:00
06961e2876
ggml webgpu: Split shared state (webgpu_context) into global state and per-thread state ( #18976 )
...
* Squashed commit of the following:
commit b3c6bf4b0450d8d452b934df27a0fb7cb53cd755
Author: Abhijit Ramesh <abhijitramesh2k@gmail.com>
Date: Mon Dec 1 18:29:00 2025 -0800
ggml webgpu: fix xielu parameter passing (#11 )
The XIELU operation was incorrectly using static_cast to convert
float parameters to uint32_t, which converted numeric values instead
of preserving IEEE 754 bit patterns. This caused incorrect values
to be interpreted by the GPU shader.
* Use reinterpret_cast to preserve float bit patterns when passing
through uint32_t params buffer
* Update WGSL shader parameter types from u32 to f32
* Re-enable XIELU support (was disabled due to numerical issues)
Fixes NMSE test failures for XIELU operation on WebGPU backend.
commit 5ca9b5e49e#10 )
* refactored pipelines
* refactored workgroup calculation
* removed commented out block of prior maps
* Clean up ceiling division pattern
---------
Co-authored-by: Neha Abbas <nehaabbas@eduroam-169-233-141-223.ucsc.edu>
Co-authored-by: Reese Levine <reeselevine1@gmail.com>
Author: James Contini <jamescontini@gmail.com>
Date: Wed Oct 29 23:13:06 2025 -0700
formatted embed wgsl and ggml-webgpu.cpp
commit e1f6baea318c70b8fecef9282c660c4cf28d7dec74c6add176362749910bcb085833375360e2852a7b09baa4aa8a6ec84374b8fc178a6ec843a5c3ae38278aaa1c9b2f88
2026-01-27 20:53:36 -08:00
f2571df8b7
ggml-zendnn : update ZenDNN git tag to main branch ( #19133 )
2026-01-28 06:21:36 +08:00
2b4cbd2834
jinja : implement mixed type object keys ( #18955 )
...
* implement mixed type object keys
* add tests
* refactor
* minor fixes
* massive refactor
* add more tests
* forgotten tuples
* fix array/object is_hashable
* correct (albeit broken) jinja responses
verified with transformers
* improved hashing and equality
* refactor hash function
* more exhausive test case
* clean up
* cont
* cont (2)
* missing cstring
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-01-27 19:50:42 +01:00
68ac3acb43
docs: Remove duplicated word on CUDA build section ( #19136 )
2026-01-27 14:48:51 +01:00
a5bb8ba4c5
CUDA: tune GLM 4.7 Flash FA kernel selection logic ( #19097 )
2026-01-27 14:28:56 +01:00
c0204a0893
ci : revert slim runner for winget ( #19129 )
2026-01-27 11:54:25 +01:00
be8890e721
ggml-cpu: aarm64: q6_K repack gemm and gemv (and generic) implementations (i8mm) #18860 ( #18888 )
...
* Boilerplate for q6_K repack
* q6_K repack to q6_Kx8 implementation
Signed-off-by: Alberto Cabrera <alberto.cabrera@liquid.ai>
* q6_K generic gemv and gemm
* wip, gemm_q6_K 8x8
* Still WIP: loading of q8s, q6h and q6l
* first working version of q6_K gemm
* Moved q6 loads outside of sb block, Unrolled inner loop
* Replaced modulo with mask
* First implementation of GEMV
* ggml_vdotq_s32 -> vdotq_s32
* Reduce width of accumulators in q6_K gemv
* Bsums instead of calc bias. Preload scales to use vget_lane. Unroll.
* Reuse scales in GEMM (same GEMV opt)
* Added todos for bsum and different qh repack
* Arch fallback
* VSLIQ for merging qh adn ql
* Removed TODO, already tested
* Apply suggestions
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Removed unused import
---------
Signed-off-by: Alberto Cabrera <alberto.cabrera@liquid.ai>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2026-01-27 11:08:10 +02:00
a83c73a18a
[CUDA] Reduce CPU-side stalls due to the CUDA command buffer being full ( #19042 )
...
* [CUDA] Reduce CPU-side stalls due to the CUDA command buffer being full
With pipeline parallelism, during prompt processing, the CPU-side CUDA command buffer gets full, stalling the CPU. Due to this, enough work doesn't get submitted to the GPU, causing bubbles in the GPU timeline.
Fix this by setting the CUDA environment variable CUDA_SCALE_LAUNCH_QUEUES to 4x to increase the command buffer size.
* Set the env variable in the CUDA backend registry allocation
* Add link to PR in code comment
* Remove warning logs and update documentation
2026-01-27 08:52:44 +02:00
fc3cdf32ce
common : clarify HTTPS build options in error message ( #19103 )
...
* common : clarify HTTPS build options in error message
This commit updates the https error message to provide clearer
instructions for users who encounter the "HTTPS is not supported" error.
The motivation for this is that it might not be clear to users that only
one of these options are needed to enable HTTPS support.
The LLAMA_OPENSSL option is also added to the message to cover all
possible build configurations.
* clarify that OpenSSL is the default for HTTPS support
2026-01-27 06:16:00 +01:00
7afdfc9b84
ggml-cpu: Enable FP16 MMA kernels on PPC ( #19060 )
2026-01-27 11:52:34 +08:00
94eeb5967c
opencl: add flattened q6_K mv ( #19054 )
...
* opencl: flatten `q6_K` and add `kernel_mul_mv_q6_K_f32_flat`
* opencl: clean up
* opencl: refactor q6_K mv - put loop body in `block_q_6_K_dot_y_flat`
* opencl: tweak the workgroup size a bit
* opencl: output 4 values per subgroup for `kernel_mul_mv_q6_K_f32_flat`
* opencl: proper alignment for q6_K
* opencl: boundary handling for flattened q6_K mv
* opencl: rename q6_K mv kernel file
* opencl: put flattened q6_K mv in its own file
* opencl: use lower k in file name
* opencl: use K in variable names
2026-01-26 19:36:24 -08:00
b0311c16d2
CUDA: fix padding of GQA to power of 2 in FA ( #19115 )
2026-01-26 23:24:58 +01:00
8f80d1b254
graph : fix nkvo offload with FA ( #19105 )
2026-01-26 20:18:34 +02:00
142cbe2ac6
ci : use new 1vCPU runner for lightweight jobs ( #19107 )
...
* use new 1vCPU runner for lightweight jobs
* pyright is too heavy, look into ty some day
use new pip-install input
2026-01-26 15:22:49 +01:00
56f3ebf38e
model : add correct type for GLM 4.7 Flash ( #19106 )
2026-01-26 11:24:30 +02:00
0c21677e43
CUDA: faster FA for GQA > 1 but not power of 2 ( #19092 )
2026-01-25 21:19:47 +01:00
0440bfd160
metal : fix recommendedMaxWorkingSetSize availability on legacy iOS/macOS ( #19088 )
...
Co-authored-by: chenbin11 <chenbin11@kuaishou.com>
2026-01-25 20:07:19 +02:00
0bf5636938
convert : yield Gemma3N custom_map tensors directly ( #19091 )
2026-01-25 18:03:34 +01:00
bcb43163ae
ggml-cpu: Use tiled FA for prompt-processing ( #19012 )
...
* ggml-cpu: Use tiled FA for prompt-processing
the FA performance is gimped on CPU on long contexts because it essentially uses a vector kernel. This PR adds a tiled FA for PP. Perf tuning for tile sizes done on a AMD EPYC single-socket 64-c machine.
* fix out of bounds for mask
* skip rows where there are all masks
* skip tile if mask is inf
* store mask in worksize
* check inf tile earlier
2026-01-25 23:25:58 +08:00
d9c6ce46f7
kv-cache : support V-less cache ( #19067 )
...
* kv-cache : support V-less cache
* cuda : better check for V_is_K_view
* cuda : improve V_is_K_view check
* graph : add comments
* hparams : refactor
2026-01-25 15:48:56 +02:00
70d860824a
convert : fix Gemma3N, GraniteMoe and Ernie4.5Moe ( #19084 )
...
* fix Gemma3N and Ernie4.5Moe
* fix GraniteMoe
2026-01-25 13:05:05 +01:00
080b161995
completion : fix prompt cache for recurrent models ( #19045 )
2026-01-25 09:12:50 +02:00
1243f93a2d
readme: update RWKV7 model links ( #19061 )
...
Signed-off-by: Molly Sophia <mollysophia379@gmail.com>
2026-01-25 09:11:19 +02:00
24bc238303
llama: fix integer type consistency in split helpers ( #18894 )
...
* llama: fix integer type consistency in split helpers
* llama: apply minor style fixes
* llama: remove trailing whitespace
2026-01-25 09:10:52 +02:00
Zheyuan Chen
Todor Boinovski
Georgi Gerganov
Aldehir Rojas
Andrew Marshall
Sigbjørn Skjæret
yulo
Vishal Singh
Aman Gupta
Neo Zhang
Ruben Ortlam
Sascha Rogmann
Daniel Bevenius
Patryk Kaminski
Oleksandr Kuvshynov
Ben Chen
Kevin Pouget
Alberto Cabrera Pérez
Nikhil Jain
David Lima
Johannes Gäßler
Gaurav Garg
shalinib-ibm
lhez
ccbinn
Molly Sophia
Jakkala Mahesh