C:\AI>vulkan\llama-server -c 32768 --port 7861 --fit off --tensor-split 12,56 -m Qwen3.5-27B-UD-Q6_K_XL.gguf
load_backend: loaded RPC backend from C:\AI\vulkan\ggml-rpc.dll
ggml_vulkan: Found 3 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 4060 Ti (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
ggml_vulkan: 1 = AMD Radeon(TM) Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
ggml_vulkan: 2 = AMD Radeon RX 7900 XTX (AMD proprietary driver) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from C:\AI\vulkan\ggml-vulkan.dll
load_backend: loaded CPU backend from C:\AI\vulkan\ggml-cpu-zen4.dll
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build: 8368 (9e2e2198b) with Clang 19.1.5 for Windows x86_64
system info: n_threads = 6, n_threads_batch = 6, total_threads = 12
system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
Running without SSL
init: using 11 threads for HTTP server
start: binding port with default address family
main: loading model
srv load_model: loading model 'Qwen3.5-27B-UD-Q6_K_XL.gguf'
llama_model_load_from_file_impl: using device Vulkan0 (NVIDIA GeForce RTX 4060 Ti) (0000:01:00.0) - 15339 MiB free
llama_model_load_from_file_impl: using device Vulkan2 (AMD Radeon RX 7900 XTX) (unknown id) - 23745 MiB free
llama_model_loader: loaded meta data with 49 key-value pairs and 851 tensors from Qwen3.5-27B-UD-Q6_K_XL.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = qwen35
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.sampling.top_k i32 = 20
llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000
llama_model_loader: - kv 4: general.sampling.temp f32 = 0.600000
llama_model_loader: - kv 5: general.name str = Qwen3.5-27B
llama_model_loader: - kv 6: general.basename str = Qwen3.5-27B
llama_model_loader: - kv 7: general.quantized_by str = Unsloth
llama_model_loader: - kv 8: general.size_label str = 27B
llama_model_loader: - kv 9: general.license str = apache-2.0
llama_model_loader: - kv 10: general.license.link str = https://huggingface.co/Qwen/Qwen3.5-2...
llama_model_loader: - kv 11: general.repo_url str = https://huggingface.co/unsloth
llama_model_loader: - kv 12: general.base_model.count u32 = 1
llama_model_loader: - kv 13: general.base_model.0.name str = Qwen3.5 27B
llama_model_loader: - kv 14: general.base_model.0.organization str = Qwen
llama_model_loader: - kv 15: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen3.5-27B
llama_model_loader: - kv 16: general.tags arr[str,3] = ["qwen3_5_moe", "unsloth", "image-tex...
llama_model_loader: - kv 17: qwen35.block_count u32 = 64
llama_model_loader: - kv 18: qwen35.context_length u32 = 262144
llama_model_loader: - kv 19: qwen35.embedding_length u32 = 5120
llama_model_loader: - kv 20: qwen35.feed_forward_length u32 = 17408
llama_model_loader: - kv 21: qwen35.attention.head_count u32 = 24
llama_model_loader: - kv 22: qwen35.attention.head_count_kv u32 = 4
llama_model_loader: - kv 23: qwen35.rope.dimension_sections arr[i32,4] = [11, 11, 10, 0]
llama_model_loader: - kv 24: qwen35.rope.freq_base f32 = 10000000.000000
llama_model_loader: - kv 25: qwen35.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 26: qwen35.attention.key_length u32 = 256
llama_model_loader: - kv 27: qwen35.attention.value_length u32 = 256
llama_model_loader: - kv 28: qwen35.ssm.conv_kernel u32 = 4
llama_model_loader: - kv 29: qwen35.ssm.state_size u32 = 128
llama_model_loader: - kv 30: qwen35.ssm.group_count u32 = 16
llama_model_loader: - kv 31: qwen35.ssm.time_step_rank u32 = 48
llama_model_loader: - kv 32: qwen35.ssm.inner_size u32 = 6144
llama_model_loader: - kv 33: qwen35.full_attention_interval u32 = 4
llama_model_loader: - kv 34: qwen35.rope.dimension_count u32 = 64
llama_model_loader: - kv 35: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 36: tokenizer.ggml.pre str = qwen35
llama_model_loader: - kv 37: tokenizer.ggml.tokens arr[str,248320] = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv 38: tokenizer.ggml.token_type arr[i32,248320] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 39: tokenizer.ggml.merges arr[str,247587] = ["─á ─á", "─á─á ─á─á", "i n", "─á t",...
llama_model_loader: - kv 40: tokenizer.ggml.eos_token_id u32 = 248046
llama_model_loader: - kv 41: tokenizer.ggml.padding_token_id u32 = 248055
llama_model_loader: - kv 42: tokenizer.chat_template str = {%- set image_count = namespace(value...
llama_model_loader: - kv 43: general.quantization_version u32 = 2
llama_model_loader: - kv 44: general.file_type u32 = 18
llama_model_loader: - kv 45: quantize.imatrix.file str = Qwen3.5-27B-GGUF/imatrix_unsloth.gguf
llama_model_loader: - kv 46: quantize.imatrix.dataset str = unsloth_calibration_Qwen3.5-27B.txt
llama_model_loader: - kv 47: quantize.imatrix.entries_count u32 = 496
llama_model_loader: - kv 48: quantize.imatrix.chunks_count u32 = 80
llama_model_loader: - type f32: 353 tensors
llama_model_loader: - type f16: 144 tensors
llama_model_loader: - type q8_0: 122 tensors
llama_model_loader: - type q6_K: 232 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q6_K
print_info: file size = 23.90 GiB (7.63 BPW)
load: 0 unused tokens
load: printing all EOG tokens:
load: - 248044 ('<|endoftext|>')
load: - 248046 ('<|im_end|>')
load: - 248063 ('<|fim_pad|>')
load: - 248064 ('<|repo_name|>')
load: - 248065 ('<|file_sep|>')
load: special tokens cache size = 33
load: token to piece cache size = 1.7581 MB
print_info: arch = qwen35
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 262144
print_info: n_embd = 5120
print_info: n_embd_inp = 5120
print_info: n_layer = 64
print_info: n_head = 24
print_info: n_head_kv = 4
print_info: n_rot = 64
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 256
print_info: n_embd_head_v = 256
print_info: n_gqa = 6
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-06
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 17408
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 40
print_info: rope scaling = linear
print_info: freq_base_train = 10000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 262144
print_info: rope_yarn_log_mul = 0.0000
print_info: rope_finetuned = unknown
print_info: mrope sections = [11, 11, 10, 0]
print_info: ssm_d_conv = 4
print_info: ssm_d_inner = 6144
print_info: ssm_d_state = 128
print_info: ssm_dt_rank = 48
print_info: ssm_n_group = 16
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 27B
print_info: model params = 26.90 B
print_info: general.name = Qwen3.5-27B
print_info: vocab type = BPE
print_info: n_vocab = 248320
print_info: n_merges = 247587
print_info: BOS token = 11 ','
print_info: EOS token = 248046 '<|im_end|>'
print_info: EOT token = 248046 '<|im_end|>'
print_info: PAD token = 248055 '<|vision_pad|>'
print_info: LF token = 198 '─è'
print_info: FIM PRE token = 248060 '<|fim_prefix|>'
print_info: FIM SUF token = 248062 '<|fim_suffix|>'
print_info: FIM MID token = 248061 '<|fim_middle|>'
print_info: FIM PAD token = 248063 '<|fim_pad|>'
print_info: FIM REP token = 248064 '<|repo_name|>'
print_info: FIM SEP token = 248065 '<|file_sep|>'
print_info: EOG token = 248044 '<|endoftext|>'
print_info: EOG token = 248046 '<|im_end|>'
print_info: EOG token = 248063 '<|fim_pad|>'
print_info: EOG token = 248064 '<|repo_name|>'
print_info: EOG token = 248065 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
load_tensors: offloading output layer to GPU
load_tensors: offloading 63 repeating layers to GPU
load_tensors: offloaded 65/65 layers to GPU
load_tensors: CPU_Mapped model buffer size = 1288.28 MiB
load_tensors: Vulkan0 model buffer size = 4034.04 MiB
load_tensors: Vulkan2 model buffer size = 19153.40 MiB
...........................................................................................
common_init_result: added <|endoftext|> logit bias = -inf
common_init_result: added <|im_end|> logit bias = -inf
common_init_result: added <|fim_pad|> logit bias = -inf
common_init_result: added <|repo_name|> logit bias = -inf
common_init_result: added <|file_sep|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max = 4
llama_context: n_ctx = 32768
llama_context: n_ctx_seq = 32768
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = auto
llama_context: kv_unified = true
llama_context: freq_base = 10000000.0
llama_context: freq_scale = 1
llama_context: n_ctx_seq (32768) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
←[0mllama_context: Vulkan_Host output buffer size = 3.79 MiB
llama_kv_cache: Vulkan0 KV buffer size = 384.00 MiB
llama_kv_cache: Vulkan2 KV buffer size = 1664.00 MiB
llama_kv_cache: size = 2048.00 MiB ( 32768 cells, 16 layers, 4/1 seqs), K (f16): 1024.00 MiB, V (f16): 1024.00 MiB
llama_memory_recurrent: Vulkan0 RS buffer size = 112.22 MiB
llama_memory_recurrent: Vulkan2 RS buffer size = 486.28 MiB
llama_memory_recurrent: size = 598.50 MiB ( 4 cells, 64 layers, 4 seqs), R (f32): 22.50 MiB, S (f32): 576.00 MiB
llama_context: pipeline parallelism enabled
llama_context: graph reuse is currently not compatible with pipeline parallelism - disabling
←[0msched_reserve: reserving ...
sched_reserve: Flash Attention was auto, set to enabled
sched_reserve: resolving fused Gated Delta Net support:
sched_reserve: fused Gated Delta Net (autoregressive) enabled
sched_reserve: fused Gated Delta Net (chunked) enabled
sched_reserve: Vulkan0 compute buffer size = 466.06 MiB
sched_reserve: Vulkan2 compute buffer size = 663.07 MiB
sched_reserve: Vulkan_Host compute buffer size = 276.08 MiB
sched_reserve: graph nodes = 3657
sched_reserve: graph splits = 3
sched_reserve: reserve took 172.95 ms, sched copies = 4
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
←[0msrv load_model: initializing slots, n_slots = 4
common_speculative_is_compat: the target context does not support partial sequence removal
←[0msrv load_model: speculative decoding not supported by this context
←[0mslot load_model: id 0 | task -1 | new slot, n_ctx = 32768
slot load_model: id 1 | task -1 | new slot, n_ctx = 32768
slot load_model: id 2 | task -1 | new slot, n_ctx = 32768
slot load_model: id 3 | task -1 | new slot, n_ctx = 32768
srv load_model: prompt cache is enabled, size limit: 8192 MiB
←[0msrv load_model: use `--cache-ram 0` to disable the prompt cache
←[0msrv load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
←[0minit: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
<think>
'
srv init: init: chat template, thinking = 1
main: model loaded
main: server is listening on http://127.0.0.1:7861
main: starting the main loop...
srv update_slots: all slots are idle
srv log_server_r: done request: GET / 127.0.0.1 200
srv params_from_: Chat format: peg-native
slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id 3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
slot update_slots: id 3 | task 0 | new prompt, n_ctx_slot = 32768, n_keep = 0, task.n_tokens = 453
slot update_slots: id 3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id 3 | task 0 | prompt processing progress, n_tokens = 449, batch.n_tokens = 449, progress = 0.991170
srv log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200
slot update_slots: id 3 | task 0 | n_tokens = 449, memory_seq_rm [449, end)
slot init_sampler: id 3 | task 0 | init sampler, took 0.04 ms, tokens: text = 453, total = 453
slot update_slots: id 3 | task 0 | prompt processing done, n_tokens = 453, batch.n_tokens = 4
slot update_slots: id 3 | task 0 | created context checkpoint 1 of 32 (pos_min = 448, pos_max = 448, n_tokens = 449, size = 149.626 MiB)
←[0msrv stop: cancel task, id_task = 0
←[0mslot release: id 3 | task 0 | stop processing: n_tokens = 515, truncated = 0
srv update_slots: all slots are idle
Name and Version
b8184 and numerous versions since. On b8183 and before, output is normal.
Using the standard release builds.
Operating systems
Windows
GGML backends
Vulkan
Hardware
RTX 4060 Ti + RX 7900 XTX
Models
Qwen3.5-27B, both Bartowski's Q6_K_M and Unsloth's UD-Q6_K_XL
Problem description & steps to reproduce
All outputs from Qwen3.5-27B are random characters.
Output is like this ever since b8184 (no thinking block or anything):
I tried several variations on the command line. The power went out several times while I was typing up the bug report, so I don't have all the exact command lines I tried, but I tried something pretty basic and automatic like
llama-server -c 32768 --port 7861and also tried manually favoring my XTX with something likellama-server --fit off --tensor-split 12,56.First Bad Commit
b8184 exactly. b8183 was fine. Even b8368 has the same broken behavior as b8184.
Relevant log output
Logs