codegemma broken on releases after v0.1.39

[evertjr]

What is the issue?

I use codegemma with continue.dev extension on vscode, it works fine on version 0.1.39. but on the last two releases it doesn't generate completions and behave very strangely in terminal.

OS

macOS

GPU

Apple

CPU

Apple

Ollama version

No response

[jmorganca]

Sorry this happened. Are you using codegemma:code?

[evertjr]

Sorry this happened. Are you using codegemma:code?

I use codegemma:2b-code-v1.1-q8_0 only for tab autocomplete, it's very good but I noticed it stopped giving suggestions.

I tried to revert to old versions of the continue extension but no luck, but as soon as I reverted Ollama to the mentioned version it worked again.

[robwilkes]

For me it seems to give no outputs.

$ ollama run codegemma:7b-code-q6_K
>>> <|fim_prefix|>import datetime
... def calculate_age(birth_year):
...     """Calculates a person's age based on their birth year."""
...     current_year = datetime.date.today().year
...     <|fim_suffix|>
...     return age<|fim_middle|>



>>> 
... <|fim_prefix|>import datetime
... def calculate_age(birth_year):
...     """Calculates a person's age based on their birth year."""
...     current_year = datetime.date.today().year
...     <|fim_suffix|>
...     return age<|fim_middle|>
... 


>>> Send a message (/? for help)

The instruct models work, but the code models seem to generate no output

[ashokgelal]

Can confirm this is still an issue even with the latest v0.1.43

OS: Apple Silicone
Model: codegemma:latest

This is the error I'm getting:

llama runner process has terminated: signal: abort trap error:unsupported op 'FLASH_ATTN_EXT'\nGGML_ASSERT: /Users/runner/work/ollama/ollama/llm/llama.cpp/ggml-metal.m:918:

[evertjr]

problem still persist in the latest update

[robwilkes]

still problematic in ollama version 0.1.45-rc4
I'm using codestral and deepseek v2 fine, but code gemma doesn't work, which is disappointing given code gemma is ranked top for FIM.

[evertjr]

I'm using granite-code for now, also good, but I still miss codegemma, it's the best for FIM.

[evertjr]

It's still broken on latest release

[knilink]

My guess is something wrong with the tokenizer which didn't treat <|fim_prefix|> etc. as special tokens

I tried the google official gguf with llama-tokenize and getting the same issue.

However decoding seems to be working fine.

Edit:
The latest llama.cpp seems to have the issue fixed. Reapply quantization to the original HuggingFace model using the latest llama.cpp(commit 1666f92) fixes the issue on my local.

For the official 1.1-2b the tokenizer config seems to be incorrected. https://huggingface.co/google/codegemma-1.1-2b/discussions/1
copy tokenizer.json and tokenzier_config.json from the 2b model as a workaround fixed the issue for me

[knilink]

did a bit more investigation can confirm llamacpp's update breaking codegemma tokenizer
tested llama.cpp@74f33ad, which ollama@v0.1.39 was using, with the old gguf model and tokenization worked fine.
so i guess need to reapply the quantization with latest llamacpp to get the issue fixed otherwise stick with old ollama.

[slashedstar]

I couldn't get the updated version of codegemma to work so I uploaded the older one: https://ollama.com/edwardz/codegemmaq6
If you want you can quantize it to other values with https://huggingface.co/spaces/ggml-org/gguf-my-repo

[evertjr]

For anyone interested I uploaded a fixed version: https://ollama.com/edwardz/codegemmaq6 If you want you can quantize it to other values with https://huggingface.co/spaces/ggml-org/gguf-my-repo

I tried to requantize but it didn't really fix the issue I was having with code completion on continue.dev extension, so I ended spinning another Ollama server on 0.1.39 version to keep using it, at least until a proper fix. Let's hope google released a new codegemma model based on Gemma 2