Eval bug: Performance in current vulkan binaries is BAD

[openconstruct]

Name and Version

Using llama-server from current build (b6615)
gives me ERNIE 4.5 21BA3B speeds of 4.x tokens per second
using the binary from july 17 2025 gives me 20.x

I renewed my OS and noticed speed was way down, so I reverted to an older version and it's full speed again.
Ubuntu 25,04, using the vulkan ubuntu binary in both cases_

Operating systems

Linux

GGML backends

Vulkan

Hardware

Ryzen 7535HS Radeon RX 6550M 32GB ram

Models

No response

Problem description & steps to reproduce

Run current b6615 ubuntu vulkan client performance is terrible ( 4.x tps)
run version from july get 20.x tps

First Bad Commit

No response

Relevant log output

idk

[0cc4m]

Please post proper benchmark runs of the current version and of the old version that was faster.

[openconstruct]

Could you point me at a resource? I am using llama-server and just reporting the difference. I will happile bench though

[0cc4m]

Sure, we have the llama-bench tool for this, you can point it at a model and it'll give you pp512 and tg128 values in a nicely-formatted way.

[openconstruct]

Hmm I dont have the ram to load them into llama-bench, I will see if I can replicate with a smaller model

[0cc4m]

If llama-server works, llama-bench should also work. If it does not, could you open another issue and put the info there?

[o0LINNY0o]

i just did the benchmark , i get lm studio speeds :

model	size	params	backend	ngl	threads	n_batch	fa	test	t/s
qwen3 4B Q8_0	4.70 GiB	4.02 B	Vulkan	99	12	512	1	pp512	969.94 ± 0.79
qwen3 4B Q8_0	4.70 GiB	4.02 B	Vulkan	99	12	512	1	tg128	60.54 ± 0.05

build: cdabeb2 (6953)

but when i use my llama-server bat file:

REM Set the working directory to the location of this script
cd /d "%~dp0"

.\llama-server.exe ^
-m "C:\llama\models\Qwen3-4B-Instruct-2507-UD-Q8_K_XL.gguf" ^
-ngl 37 ^
-c 16384 ^
--flash-attn on ^
--threads 12 ^
--jinja ^
--chat-template-file "C:\llama\QWEN3_Instruct.jinja" ^
--port 8080 ^
--temp 0.6 ^
--top-p 0.95 ^
--top-k 40 ^
--min-p 0.0 ^
--no-mmap ^
--mlock ^
--context-shift ^
--rope-scaling yarn --rope-scale 4.0 --yarn-ext-factor 1.0 --yarn-attn-factor 1.0 --yarn-beta-slow 1.0 --yarn-beta-fast 32.0 ^
--cache-type-k q5_1 ^
--cache-type-v q5_1

PAUSE

I only get at best 22 t/s , with a simple question like "what is a kangaroo" and the CPU usage goes 100%

[0cc4m]

Then you have a configuration difference between LMStudio and your script. Disable parameters and see how they influence the speed. For example q5_1 kv cache is unusual and may not work with flash attention.

[openconstruct]

I tried llama-bench but ir reported about 9 tps for both, but i consistently get over 20 tps with the older builds, here is the same prompt on the same model minutes apart with a version from yesterday vs a version from august. Both are running on windows 11 using vulkan binaries, I have seen the same speed differences on liuc also.

I only use llama-server so it's possible this bug is not present in bench or cli.

Yesterday:
slot print_timing: id 3 | task 0 |
prompt eval time = 6663.74 ms / 12 tokens ( 555.31 ms per token, 1.80 tokens per second)
eval time = 49073.28 ms / 425 tokens ( 115.47 ms per token, 8.66 tokens per second)
total time = 55737.02 ms / 437 tokens
slot release: id 3 | task 0 | stop processing: n_tokens = 436, truncated = 0
srv update_slots: all slots are idle

August:
prompt eval time = 513.81 ms / 12 tokens ( 42.82 ms per token, 23.36 tokens per second)
eval time = 26885.18 ms / 599 tokens ( 44.88 ms per token, 22.28 tokens per second)
total time = 27398.99 ms / 611 tokens
srv update_slots: all slots are idle

[o0LINNY0o]

Then you have a configuration difference between LMStudio and your script. Disable parameters and see how they influence the speed. For example q5_1 kv cache is unusual and may not work with flash attention.

well this was it for me , TY ... went to q8_0 and it worked