# Vulkan Chunked Gated Delta Net (GDN) — Performance & Development Notes

PR #20377 — First chunked parallel GDN implementation on any GPU shader backend.

## Architecture

Three-stage chunked parallel decomposition (matches FLA/NVlabs reference implementations):

1. **Intra-chunk** (`gated_delta_net_chunk_intra.comp`) — Builds attention matrix A, computes W/U via WY representation. Outputs g_cumsum and total chunk decay.
2. **Inter-chunk** (`gated_delta_net_chunk_inter.comp`) — Sequential across chunks, parallel across state columns. State update: `S_next = exp(g_total) * S + K_gated^T @ v_corrected`.
3. **Output** (`gated_delta_net_chunk_output_cm1.comp`) — Coopmat GEMM kernel. Computes `A_decayed[64x64] @ vnew[64x128]` using VK_KHR_cooperative_matrix (f16 inputs, f32 accumulation).

Chunk size: C=64 tokens. State dimensions: S_K=S_V=128. Pipeline: d128 non-KDA configs only.

## Development History

### Phase 1: Infrastructure (PR #20334, merged)
- Autoregressive GDN Vulkan shader — single-token sequential processing
- PP-512: 165 t/s, TG-128: 21.2 t/s on 890M (16 CU)
- 13/13 backend-ops tests

### Phase 2: Graph-level chunked ops (PR #20340, merged)
- Chunked op decomposition at the GGML graph level
- Feeds autoregressive shader more efficiently
- PP-512: 165 → 220 t/s (+30.3%) — this gain is already in master

### Phase 3: Vulkan chunked shaders (PR #20377, this PR)
- Three new compute shaders for intra/inter/output stages
- Initial scalar output kernel — functional but dispatch overhead made it slower than autoregressive on 16 CU
- Threshold gating: chunked path activates only when beneficial

### Phase 4: Coopmat output kernel
- Replaced scalar output with VK_KHR_cooperative_matrix GEMM
- f16 shared memory for A_decayed and vnew, f32 accumulation via coopmat
- 4-phase architecture: QK^T via coopmat → decay mask → vnew staging → A_decayed @ vnew GEMM
- Numerically stable: direct `exp(g_i - g_j)` per element (no factorization — factorized approach caused PPL regression to 20.06)
- 16/16 backend-ops tests pass

### Abandoned Approaches
- **Factorized exp with g_max**: `exp(g_max - gcum[j])` amplified vnew, caused catastrophic cancellation. PPL 20.06 vs 13.46 baseline.
- **Scoped register split**: Attempted to reduce VGPR pressure via scope boundaries. RADV compiler ignores scope for register allocation — no measurable difference.

## Current Performance

Hardware: AMD Radeon 890M (RDNA3.5, 16 CU, 64KB LDS/CU, warp 64, KHR_coopmat)
Model: Qwen3-Coder-Next-REAM Q4_K_M (60.33B params, 34.21 GiB)

### Throughput (chunked coopmat, GDN_CHUNK_THRESHOLD=2)

| Test | t/s |
|------|-----|
| PP-512 | 217.55 ± 1.41 |
| PP-1024 | 219.84 ± 4.00 |
| PP-2048 | 216.89 ± 1.94 |
| TG-128 | 21.76 ± 0.06 |

### Autoregressive vs Chunked Comparison

| Test | Autoregressive | Chunked coopmat | Delta |
|------|---------------|-----------------|-------|
| PP-512 | 225.68 ± 3.00 | 217.55 ± 1.41 | -3.6% |
| PP-1024 | 229.63 ± 4.39 | 219.84 ± 4.00 | -4.3% |
| PP-2048 | 230.88 ± 1.44 | 216.89 ± 1.94 | -6.1% |
| TG-128 | 21.29 ± 0.03 | 21.76 ± 0.06 | +2.2% |

On 16 CU, autoregressive is 3.6-6.1% faster for PP due to lower dispatch overhead. Note autoregressive PP improves from 512→2048 while chunked stays flat — the gap widens on small hardware but the scaling characteristics favor chunked on wider hardware.

GDN kernel time comparison (PP-512):
- Autoregressive: 36 × 1,150 us = 41 ms (1.8% of total)
- Chunked (3 dispatches): 36 × 5,173 us = 186 ms (7.9% of total)

The chunked path's 3-dispatch overhead (intra + inter + output) accounts for the per-kernel cost difference, but end-to-end impact is only 3.6-6.1% since GDN is a small fraction of total wall time on this MoE model.

### Perplexity Validation (WikiText-2, 299K tokens)

| Context | Chunked coopmat | f32 baseline | Delta |
|---------|----------------|--------------|-------|
| 512 (584 chunks) | 13.52 ± 0.11 | 13.46 | +0.06 |
| 4096 (73 chunks) | 10.18 ± 0.08 | 10.15 | +0.03 |

Both within error bars. Chunked coopmat path is numerically lossless.

### Per-Kernel Timing (GGML_VK_PERF_LOGGER, PP-512)

```
GATED_DELTA_NET: 36 × 5173 us = 186 ms (7.9% of 2.35s total)
FLASH_ATTN_EXT:  12 × 783 us  = 9.4 ms (0.4% of 2.35s total)
```

GDN is 7.9% of PP-512 wall time on this MoE-heavy model. MUL_MAT and MoE routing dominate the remaining 92%.

## Scaling Analysis

### Why flat PP scaling matters
PP-512/1024/2048 all within ±2 t/s. The chunked architecture processes fixed-size 64-token chunks — adding more tokens adds more chunks at constant cost each. Autoregressive dispatches scale linearly with token count (36 layers × N tokens = 36N sequential dispatches).

### Why 16 CU doesn't show the crossover
- Chunked output kernel dispatches 3 shaders (intra + inter + output) vs 1 for autoregressive
- Each shader has launch overhead (~10-20 us) that dominates on small hardware
- The 64×64 @ 64×128 coopmat GEMM in the output kernel can't saturate 16 CUs
- On 40+ CU hardware (e.g., Strix Halo 8060S, discrete GPUs), the matmul-heavy chunked path has more headroom

### GDN share grows with model density
On Qwen3-Next (384-expert MoE), GDN is only 8% of wall time. On GDN-dense architectures with fewer/no MoE layers, GDN's share would be 30-40%+, making the chunked optimization proportionally more impactful.

## Key Files

| File | Purpose |
|------|---------|
| `vulkan-shaders/gated_delta_net.comp` | Autoregressive kernel |
| `vulkan-shaders/gated_delta_net_chunk_intra.comp` | Intra-chunk (A matrix, WY) |
| `vulkan-shaders/gated_delta_net_chunk_inter.comp` | Inter-chunk (state update) |
| `vulkan-shaders/gated_delta_net_chunk_output.comp` | Original scalar output |
| `vulkan-shaders/gated_delta_net_chunk_output_cm1.comp` | Coopmat GEMM output |
| `ggml-vulkan.cpp:10409` | GDN_CHUNK_THRESHOLD (dispatch gating) |

## Test Commands

```bash
# Backend ops tests
./build/bin/test-backend-ops -b Vulkan0 -o GATED_DELTA_NET

# Benchmark
./build/bin/llama-bench -m <model> -ngl 99 -fa 1 -n 128 -p 512 --output md

# Perf logger
GGML_VK_PERF_LOGGER=1 ./build/bin/llama-bench -m <model> -ngl 99 -fa 1 -n 128 -p 512 -r 3 --output md

# Perplexity
./build/bin/llama-perplexity -m <model> -ngl 99 -fa 1 --ctx-size 4096 -f data/wikitext-2-raw/wiki.test.raw
```