Eval bug: Qwen3.5-35B-A3B produces garbage output on Vulkan backend for specific prompts (Intel Arc MTL iGPU), correct on CPU

[spf1983]

Summary

On the Intel Arc Graphics (Meteor Lake, Xe-LPG) integrated GPU with Mesa ANV 25.2.8, the qwen3_5moe / Qwen3.5-35B-A3B Q4_K_M model produces valid output for short, high-frequency greedy completions (e.g. "The capital of France is" → "Paris"), but emits deterministic repeating-punctuation garbage (!"!"""...) for other prompts — notably anything with "Fourier transform" — while the same model + same quant + same prompt on CPU (-ngl 0) works correctly. The GGML_VK_DISABLE_F16=1 workaround from #18969 does not fix this.

Accompanying symptom: when the prompt reproduces the bug, prompt-eval drops from ~13 tok/s to ~0.48 tok/s (same slow path that CPU takes), strongly suggesting a specific kernel has a correctness failure / fallback for these token sequences.

This may be related to #19957 (SSM kernels), #20354 (Gated Delta Net shader), #20610 (Qwen3.5-27B garbage since b8184), or #18969 (f16 acc), but is distinct from all of them in behavior. Reporting separately in case it's a residual GDN / hybrid-memory path bug.

Name and Version

version: 1 (e21cdc1)
built with GNU 13.3.0 for Linux x86_64

Built from commit e21cdc1 (tip of master at the time of testing, which already contains ssm_conv.comp, ssm_scan.comp, gated_delta_net.comp shaders).

Build command:

cmake -B build -DGGML_VULKAN=ON -DBUILD_SHARED_LIBS=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc) --target llama-cli llama-bench llama-simple

Operating systems

Linux

GGML backends

Vulkan

Hardware

• CPU: Intel Core Ultra 5 125H (Meteor Lake, 18 threads)
• iGPU: Intel Arc Graphics (MTL), PCI [8086:7d55], Xe-LPG
  • Kernel driver: i915 (xe also loaded)
  • Mesa 25.2.8-0ubuntu0.24.04.1 (ANV / Intel open-source Mesa driver)
  • Vulkan API 1.4.318, driver 25.2.8
  • uma: 1, fp16: 1, bf16: 0, warp size 32, no matrix cores
• RAM: 32 GiB DDR5-4800 (2×16G)
• Kernel: 6.8.1-t6-generic (Ubuntu 24.04 derivative)
• /dev/dri/renderD128 accessible (user in render + video groups)

vulkaninfo --summary excerpt:

GPU0:
  apiVersion         = 1.4.318
  driverVersion      = 25.2.8
  vendorID           = 0x8086
  deviceID           = 0x7d55
  deviceType         = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
  deviceName         = Intel(R) Arc(tm) Graphics (MTL)
  driverID           = DRIVER_ID_INTEL_OPEN_SOURCE_MESA
  driverInfo         = Mesa 25.2.8-0ubuntu0.24.04.1

Models

unsloth/Qwen3.5-35B-A3B-GGUF, file Qwen3.5-35B-A3B-Q4_K_M.gguf

• Size: 22016023168 bytes (20.49 GiB)
• SHA256: 3b46d1066bc91cc2d613e3bc22ce691dd77e6f0d33c9060690d24ce6de494375
• llama.cpp architecture label: qwen35moe 35B.A3B Q4_K - Medium, 34.66 B params
• Loader reports use of llama_memory_recurrent, fused Gated Delta Net (autoregressive/chunked) enabled

Reproduction

Step 1 — Load the model and confirm Vulkan full offload works

$ build/bin/llama-bench -m Qwen3.5-35B-A3B-Q4_K_M.gguf -p 64 -n 32 -ngl 99 -fa 1
ggml_vulkan: 0 = Intel(R) Arc(tm) Graphics (MTL) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | bf16: 0
| model                              |    size |  params | backend | ngl | fa | test | t/s              |
| qwen35moe 35B.A3B Q4_K - Medium    | 20.49 GiB | 34.66 B | Vulkan |  99 |  1 | pp64 | 28.32 ± 0.95     |
| qwen35moe 35B.A3B Q4_K - Medium    | 20.49 GiB | 34.66 B | Vulkan |  99 |  1 | tg32 |  6.02 ± 0.01     |

Step 2 — Working case (Vulkan, -ngl 99)

$ build/bin/llama-simple -m Qwen3.5-35B-A3B-Q4_K_M.gguf -n 40 -ngl 99 "The capital of France is"

Output (correct):

The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
...

Perf: prompt 13.04 tok/s, gen 6.03 tok/s.

Also working on Vulkan:

• "北京的著名景点有" → "故宫、天坛、颐和园、长城..."
• "def fibonacci(n):" → valid Python continuation
• "Quantum computing is" → coherent English prose

Step 3 — Failing case (Vulkan, -ngl 99), deterministic garbage

$ build/bin/llama-simple -m Qwen3.5-35B-A3B-Q4_K_M.gguf -n 40 -ngl 99 "Question: What is the Fourier transform? Answer:"

Output (garbage, reproduced 3+ times):

Question: What is the Fourier transform? Answer:!"!!!!!!!!!"""""""""""""""""""""""""""""

Perf:

• prompt eval: 0.48 tok/s / 2075 ms per token (vs ~13 tok/s on the working prompt — a ~27× regression)
• gen eval: 6.00 tok/s (normal)

Also fails on Vulkan with the same garbage pattern:

• "用一句话解释什么是傅里叶变换:" (Chinese)
• "Question: What is the Fourier transform? Answer in one sentence."

Step 4 — Control: CPU (-ngl 0) with the same failing prompt

$ build/bin/llama-simple -m Qwen3.5-35B-A3B-Q4_K_M.gguf -n 20 -ngl 0 "Question: What is the Fourier transform? Answer:"

Output (correct):

Question: What is the Fourier transform? Answer: The Fourier transform is a mathematical operation that decomposes a function into its constituent frequencies. It is used

Perf: 0.39 tok/s decode (expected for 35B-A3B on 18-thread CPU).

→ Same binary, same model, same quant, same prompt — only the -ngl value differs. CPU produces correct output; Vulkan produces garbage.

Attempted workarounds (all failed to fix this bug)

GGML_VK_DISABLE_F16=1

$ GGML_VK_DISABLE_F16=1 build/bin/llama-simple -m ... -n 40 -ngl 99 "Question: What is the Fourier transform? Answer:"
Question: What is the Fourier transform? Answer:!!!!!!!!!!!!!!!""""""""!!!!"""""""""""""

• Device caps correctly report fp16: 0 after setting the env var, confirming it took effect
• Bench shows pp64 jumps from 28.32 → 41.45 tok/s (46% gain) and fp16 is now 0 — so the env var is being honored
• The "working" prompts still work; the failing prompts still fail with the same pattern

Additional observations

1. The bug pattern is deterministic and prompt-specific — same prompt produces same garbage across runs.
2. The garbage is always !/"/# or similar punctuation tokens (lowest-index ASCII-like vocab IDs), suggesting logits collapse toward a fixed set of tokens — consistent with NaN/Inf or very negative activations propagating through softmax.
3. The ~27× prompt-eval slowdown on failing prompts (without workaround) is notable — it implies the compute graph for these prompts is taking a different (slow) path, possibly a CPU fallback for a specific op that also corrupts hidden state on the GPU↔CPU boundary.
4. With GGML_VK_DISABLE_F16=1, the slow prompt-eval path still happens (0.27 tok/s for Fourier prompt), so it's not only f16acc — some kernel is still falling back or stalling.
5. vulkaninfo shows 3 GPU entries for the same device (GPU0/GPU1/GPU2 with identical UUIDs); llama-cli --list-devices correctly shows a single Vulkan0: Intel(R) Arc(tm) Graphics (MTL) (23711 MiB, 21339 MiB free).
6. ssm_conv.comp, ssm_scan.comp, and gated_delta_net.comp compute shaders are all present in ggml/src/ggml-vulkan/vulkan-shaders/ in the build tree.

Why I think this is a fresh issue, not a duplicate

• Misc. [Bug]: Vulkan Backend - NaN on Intel iGPU when injecting Embeddings (f16 acc overflow) #18969 (f16 acc NaN on embeddings): the GGML_VK_DISABLE_F16 workaround explicitly fixed that issue. It does not fix this one.
• Feat/Bug: Vulkan backend missing ggml_ssm_conv / ggml_ssm_scan kernels — Qwen3.5-35B-A3B (qwen3_5moe) CPU-only on Vulkan #19957 (missing SSM kernels for qwen3_5moe): reported resolved by recompiling to a version with the shaders — our build has the shaders and model does load + run, but with silent correctness failure on specific prompts.
• Vulkan: missing GATED_DELTA_NET compute shader; ROCm/HIP: fused kernel underperforms on RDNA 3.5 (gfx1151) #20354 (GDN Vulkan compute shader): closed as dup; my build has gated_delta_net.comp.
• Eval bug: garbage output for Qwen3.5-27B on Vulkan since b8184 #20610 (Qwen3.5-27B garbage on Vulkan since b8184): similar symptom family, different model, unresolved; might share root cause.

Attachments / follow-ups

• Full llama-simple stderr of the failing run (available on request)
• Full llama-bench output with and without GGML_VK_DISABLE_F16=1 (shown in repro section)
• Happy to provide additional logs, run with LLAMA_LOG_DEBUG=1, bisect the specific compute op (e.g. dump tensor state at each layer to narrow down which kernel corrupts output), or test patches on this hardware.

[spf1983]

Follow-up: root cause appears to be Intel userspace graphics libraries, not llama.cpp

Update after further investigation — this issue appears to be fully fixed on my system without any llama.cpp code change. The fix is upgrading the Intel userspace graphics stack that Mesa ANV links against.

What fixed it

While investigating the SYCL backend as a workaround, I installed the Intel Graphics apt repo:

https://repositories.intel.com/gpu/ubuntu noble client

This upgraded several Intel-specific userspace libraries that Mesa ANV shares with the Intel Compute Runtime:

Package	Before (Ubuntu 24.04 universe)	After (Intel Graphics repo)
libigdgmm12 (Intel Graphics Memory Manager)	22.3.17+ds1-1ubuntu1	22.5.2-1018~24.04
libigc1 (Intel Graphics Compiler)	1.0.15468.25-2ubuntu0.1	1.0.17791.16-1032~24.04
libigdfcl1	1.0.15468.25-2ubuntu0.1	1.0.17791.16-1032~24.04
intel-opencl-icd	23.43.27642.40-1ubuntu3	24.39.31294.20-1032~24.04
libze-intel-gpu1	(not installed)	24.39.31294.20-1032~24.04
libze1 (level-zero loader)	1.16.1-1build1	1.17.44.0-1022~24.04

mesa-vulkan-drivers was unchanged at 25.2.8-0ubuntu0.24.04.1 on both sides — only the Intel-specific libraries that Mesa ANV links at runtime changed.

Verification

Same llama.cpp build e21cdc1, same model (unsloth Qwen3.5-35B-A3B-Q4_K_M.gguf, sha256 3b46d1066bc91cc2d613e3bc22ce691dd77e6f0d33c9060690d24ce6de494375), same -ngl 99 -fa 1 command, same hardware (Intel Core Ultra 5 125H / Meteor Lake / 8086:7d55 / i915).

Before the Intel library upgrade — 3 deterministic reproductions of garbage:

$ llama-simple ... "Question: What is the Fourier transform? Answer:"
Question: What is the Fourier transform? Answer:!"!!!!!!!!!"""""""""""""""""""""""""""""

After the Intel library upgrade — correct output via llama-server /completion:

$ curl -s localhost:18080/completion -d '{"prompt":"Question: What is the Fourier transform? Answer:","n_predict":30,"temperature":0}'
{"content": " The Fourier transform is a mathematical operation that decomposes a function into its constituent frequencies. It is used to analyze the frequency content of a signal or function..."}

I re-ran 10 prompts (the originally failing "Fourier Q: A:" pattern in English and Chinese, plus controls with other Q: A: content, plus prompts that isolate question mark / colon / content separately). All 10 produce coherent, correct output. Previously reproducible failures no longer occur.

Probable root cause

The most likely culprits are libigdgmm12 (22.3.17 → 22.5.2) or libigc1 (1.0.15468 → 1.0.17791). Both span roughly a year of Intel graphics stack development and contain significant fixes for Meteor Lake / Xe-LPG. libigdgmm12 in particular handles GPU buffer allocation and is linked by both Mesa ANV and the Intel Compute Runtime. A latent memory-layout or tiling bug in the old version would plausibly manifest as corrupted hidden state during GDN / SSM autoregressive state updates on specific token patterns (explaining the prompt-dependent nature of the original failure).

This would mean the ggml-vulkan GDN / SSM compute shaders themselves are correct — the garbage output was caused by a bug in how buffers they touched were being managed at a lower level, which happened to trigger for the specific token embeddings that appeared in "Fourier transform" prompts.

Workaround for anyone else hitting this on Ubuntu 24.04 Intel iGPU

curl -fsSL https://repositories.intel.com/gpu/intel-graphics.key \
  | sudo gpg --dearmor -o /usr/share/keyrings/intel-graphics.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu noble client" \
  | sudo tee /etc/apt/sources.list.d/intel-gpu-noble.list
sudo apt update
sudo apt install -y libigdgmm12 libigc1 libigdfcl1 intel-opencl-icd libze1

(No llama.cpp rebuild required. mesa-vulkan-drivers does not need to be touched.)

Should this issue stay open?

From llama.cpp's side, this is arguably not a bug in the project — the fix lives in Intel's userspace stack. But it may be worth keeping open as a known gotcha, or adding a note to the Vulkan backend README saying that Ubuntu 24.04 universe's libigdgmm12 22.3.17 / libigc1 1.0.15468 are too old for stable Vulkan inference on Meteor Lake (and probably Arrow Lake / Lunar Lake) and that users should install from Intel's apt repo.

Happy to close this once a maintainer confirms they want it tracked or not.

[spf1983]

Correction: the previous comment's root cause claim was wrong

I ran a decisive downgrade experiment to confirm whether the Intel userspace libraries were actually responsible for the fix, and the result invalidates my previous conclusion.

Experiment

Downgraded all the libraries I had previously credited with fixing the bug, back to the exact Ubuntu 24.04 universe versions:

sudo apt install -y --allow-downgrades \
  libigdgmm12=22.3.17+ds1-1ubuntu1 \
  libigc1=1.0.15468.25-2ubuntu0.1 \
  libigdfcl1=1.0.15468.25-2ubuntu0.1 \
  intel-opencl-icd=23.43.27642.40-1ubuntu3 \
  libze1=1.16.1-1build1
sudo apt remove -y libze-intel-gpu1

Confirmed versions back to Ubuntu universe baseline. Then ran the exact command that previously produced garbage, three times back-to-back (fresh llama-simple processes each time, no llama-server reuse):

===downgrade_run1===
Question: What is the Fourier transform? Answer: The Fourier transform is a mathematical operation that decomposes a function into its constituent frequencies. It is used to analyze the frequency content of a signal or function

===downgrade_run2===
Question: What is the Fourier transform? Answer: The Fourier transform is a mathematical operation that decomposes a function into its constituent frequencies. It is used to analyze the frequency content of a signal or function

===downgrade_run3 (control)===
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.

All three runs produce correct, bit-identical, deterministic output, with the old libraries. The fix is not in libigdgmm / libigc / libigdfcl / intel-opencl-icd / libze1.

What I actually know

1. At the time of filing this issue, Question: What is the Fourier transform? Answer: reproducibly produced punctuation-garbage output on -ngl 99 Vulkan, with prompt eval stuck around 0.48 tok/s (vs ~13 tok/s for passing prompts). I have multiple reproductions with the same command line.
2. After some sequence of actions — installing oneAPI, running SYCL tests, multiple drop_caches + swapoff/swapon cycles, several OOM kills (caught in dmesg), then installing the Intel Graphics apt repo — the bug stopped reproducing.
3. I then downgraded the Intel userspace libs back and the bug still does not reproduce.
4. strings libvulkan_intel.so has no references to libigdgmm/libigc/libigdfcl/intel_opencl, and ldd shows no link-time dependency. So Mesa ANV genuinely does not use these libraries, which is consistent with the downgrade experiment showing they are not the cause.

New best guess

This points at something stateful that got cleared between the failing and passing runs, most likely one of:

• i915 driver GEM/BO state — multiple OOM kills and process churn between those sessions would have forced i915 to reclaim all GEM objects from the old processes.
• Shader / pipeline cache — Mesa ANV maintains an on-disk shader cache (typically ~/.cache/mesa_shader_cache). If a corrupted entry was cached during an early run, it could have affected subsequent runs until the cache was invalidated/evicted. I didn't verify whether my actions (drop_caches, etc.) touched this specifically.
• Genuinely flaky / non-deterministic — the earlier reproductions may have shared a latent non-deterministic condition (e.g., buffer alignment depending on allocation order, race in a kernel launch) that has not been re-hit since.

I no longer have a reliable repro for this bug, and I cannot confirm it was ever deterministic in the way I originally described.

Practical recommendation

Please feel free to close this issue unless you think the symptom pattern (prompt-dependent punctuation garbage + ~27× prompt-eval slowdown on Qwen3.5-35B-A3B / qwen3_5moe hybrid on Intel iGPU Mesa ANV) is worth keeping as a tracker. If anyone else hits the same symptom, useful things to try first are:

• Clear Mesa shader cache: rm -rf ~/.cache/mesa_shader_cache_db ~/.cache/mesa_shader_cache
• Kill any lingering GPU processes and re-run
• Reboot

Apologies for the incorrect root-cause claim in the previous comment — the library-upgrade correlation turned out to be spurious.