fix wrong template in GLM4-0414

[matteoserva]

Reopening #13099 because I broke my repo with a wrong git command and it automatically closed the PR.

@ngxson

• I moved the check for GLM4 above so it gets higher priority.

Instead of changing BOS, I think what should be done is to set add_add_bos(False) to prevent tokenize function from adding BOS to the sentence

I am not sure if I can do that. add_bos_token is automatically set to false by the HF model. I am forced to set it to PAD token because otherwise it would break the jinja templating system ./llama-server --jinja.
The jinja code ignores the add_bos_token value and removes any BOS token it encounters.

This is the relevant code block.

llama.cpp/common/chat.cpp

Line 651 in e291450

if (string_starts_with(result, tmpl.bos_token())) {

If I don't set the BOS token it would break the /props endpoint

ORIGINAL:

GLM4-0414 models were using the wrong legacy template, leading to a missing `[gMASK]<sop>` preamble.
The old code was returning `LLM_CHAT_TEMPLATE_GLMEDGE`

As a workaround you needed to launch `llama-server` with `--chat-template chatglm4`

After the patch the gguf should be regenerated or edited manually.

[ngxson]

#13099 (comment)

[matteoserva]

I'm not sure I can do that.

• add_bos_token is already set to False by the HF model and conversion script.
• The jinja templating system always removes the bos token, ignoring add_bos_token. So the prefix is removed and never added again.
• If I remove the bos token altogether, the /props endpoint crashes because it expects it

This is the relevant jinja template code. reached when ./llama-server --jinja

llama.cpp/common/chat.cpp

Line 651 in e291450

if (string_starts_with(result, tmpl.bos_token())) {

[ngxson]

if the BOS is not added, then what is the reason behind changing it from [gMASK] to <|endoftext|>? it shouldn't crash /props, right?

Nvm I see what you mean

[Mushoz]

Do GGUFs have to be regenerated for these fixes to apply, or can I continue using the GGUF I already have?

[matteoserva]

Do GGUFs have to be regenerated for these fixes to apply, or can I continue using the GGUF I already have?

The GGUF should be regenerated or edited manually.

[Dampfinchen]

Hello! Sorry, noob here. What changes would I have to make to my existing GGUF? I've looked at the template and I'm just confused.

[matteoserva]

Hello! Sorry, noob here. What changes would I have to make to my existing GGUF? I've looked at the template and I'm just confused.

you only have to replace the bos token kv parameter from [gMask] to <|endoftext|> in the GGUF, then use a updated llama.cpp

[Ph0rk0z]

kv parameter? Do you mean edit the bos token ID in the metadata? or somewhere else. And I assume this means I was testing the model with bos token listed twice, even in text completion?

I wish you could simply choose to add the bos or not in text completions from the front end. I'm under the impression that it's added anyway.

[matteoserva]

kv parameter? Do you mean edit the bos token ID in the metadata? or somewhere else. And I assume this means I was testing the model with bos token listed twice, even in text completion?

I wish you could simply choose to add the bos or not in text completions from the front end. I'm under the impression that it's added anyway.

The command to enable/disable the bos token is --override-kv tokenizer.ggml.add_bos_token=bool:false.
In GLM the value add_bos_token is not set and it defaults to false.

The problem is that the jinja templating system ignores add_bos_token and breaks the prompt. The fix for a old GGUF is simply to change the bos token (tokenizer.ggml.bos_token_id) from [gMASK] to <|endoftext|> and run it with an updated llama.cpp