happyz
/
llama.cpp
mirror of https://github.com/ggerganov/llama.cpp.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
llama.cpp/tools/mtmd/models
History
Stephen Cox 547765a93e
mtmd: add Gemma 4 audio conformer encoder support (#21421) 
* mtmd: add Gemma 4 audio conformer encoder support

Add audio processing for Gemma 4 E2B/E4B via a USM-style Conformer.

Architecture:
- 12-layer Conformer: FFN → Self-Attention → Causal Conv1D → FFN → Norm
- Subsampling Conv Projection: 2x Conv2D(stride=2) with LayerNorm
- Full self-attention with sinusoidal RPE and sliding window mask (24)
- Logit softcapping at 50.0, ClippableLinear clamping
- Output: 1024 → 1536 → RMSNorm → multimodal embedder

Mel preprocessing (dedicated mtmd_audio_preprocessor_gemma4a):
- HTK mel scale, 128 bins, magnitude STFT, mel_floor=1e-3
- Standard periodic Hann window (320 samples), zero-padded to FFT size
- Semicausal left-padding (frame_length/2 samples)
- Frame count matched to PyTorch (unfold formula)
- No pre-emphasis, no Whisper-style normalization
- Mel cosine similarity vs PyTorch: 0.9998

Key fixes:
- Tensor loading dedup: prevent get_tensor() from creating duplicate
  entries in ctx_data. Fixed with std::set guard.
- ClippableLinear clamp_info loading moved after per-layer tensors.
- Sliding window mask (24 positions) matching PyTorch context_size.
- Skip Whisper normalization for Gemma4 mel output.

Tested on E2B and E4B with CPU and Vulkan backends.
Transcribes: "Glad to see things are going well and business is starting
to pick up" (matching ground truth).

Ref: #21325
 		2026-04-12 14:15:26 +02:00
..
cogvlm.cpp mtmd: add clip_graph::build_mm() (#20751) 2026-03-19 13:11:39 +01:00
conformer.cpp mtmd: add clip_graph::build_mm() (#20751) 2026-03-19 13:11:39 +01:00
deepseekocr.cpp mtmd: Add DeepSeekOCR Support (#17400) 2026-03-25 19:57:40 +01:00
dotsocr.cpp mtmd: support dots.ocr (#17575) 2026-04-09 12:16:38 +02:00
gemma4a.cpp mtmd: add Gemma 4 audio conformer encoder support (#21421) 2026-04-12 14:15:26 +02:00
gemma4v.cpp model, mtmd: fix gguf conversion for audio/vision mmproj (#21309) 2026-04-02 17:10:32 +02:00
glm4v.cpp mtmd: Add DeepSeekOCR Support (#17400) 2026-03-25 19:57:40 +01:00
hunyuanocr.cpp model : add HunyuanOCR support (#21395) 2026-04-05 23:32:14 +02:00
internvl.cpp clip: move model cgraphs into their own files (#17965) 2025-12-12 21:14:48 +01:00
kimik25.cpp model: Add Kimi-K2.5 support (#19170) 2026-02-11 16:47:30 +01:00
kimivl.cpp clip: move model cgraphs into their own files (#17965) 2025-12-12 21:14:48 +01:00
llama4.cpp mtmd: add clip_graph::build_mm() (#20751) 2026-03-19 13:11:39 +01:00
llava.cpp mtmd: add clip_graph::build_mm() (#20751) 2026-03-19 13:11:39 +01:00
minicpmv.cpp mtmd: add clip_graph::build_mm() (#20751) 2026-03-19 13:11:39 +01:00
mobilenetv5.cpp mtmd: add clip_graph::build_mm() (#20751) 2026-03-19 13:11:39 +01:00
models.h mtmd: add Gemma 4 audio conformer encoder support (#21421) 2026-04-12 14:15:26 +02:00
nemotron-v2-vl.cpp mtmd : Add Nemotron Nano 12B v2 VL support (#19547) 2026-02-14 14:07:00 +01:00
paddleocr.cpp model: Add PaddleOCR-VL model support (#18825) 2026-02-19 17:05:25 +01:00
pixtral.cpp mtmd: add clip_graph::build_mm() (#20751) 2026-03-19 13:11:39 +01:00
qwen2vl.cpp mtmd: add clip_graph::build_mm() (#20751) 2026-03-19 13:11:39 +01:00
qwen3vl.cpp mtmd: add clip_graph::build_mm() (#20751) 2026-03-19 13:11:39 +01:00
siglip.cpp mtmd: Add DeepSeekOCR Support (#17400) 2026-03-25 19:57:40 +01:00
step3vl.cpp model : support step3-vl-10b (#21287) 2026-04-08 09:51:31 +02:00
whisper-enc.cpp mtmd : add MERaLiON-2 multimodal audio support (#21756) 2026-04-11 14:15:48 +02:00
youtuvl.cpp mtmd: add clip_graph::build_mm() (#20751) 2026-03-19 13:11:39 +01:00