Eval bug: LFM2-VL giving different outputs on large images.

[SmartestWashingMachine]

Name and Version

load_backend: loaded RPC backend from C:\Users\metal\Downloads\llama-b7066-bin-win-cpu-x64\ggml-rpc.dll
load_backend: loaded CPU backend from C:\Users\metal\Downloads\llama-b7066-bin-win-cpu-x64\ggml-cpu-alderlake.dll
version: 7066 (1568d13)
built with clang version 19.1.5 for x86_64-pc-windows-msvc

Operating systems

Windows

GGML backends

CPU, CUDA

Hardware

Intel i5-12500H + RTX 3050M

Models

LFM2-VL-1.6B-Q8_0.gguf
mmproj-LFM2-VL-1.6B-Q8_0.gguf

Problem description & steps to reproduce

When running LFM2-VL-1.6B on an image with one side having a dimension of 1024px, the outputs become vastly different than when compared to PyTorch. Not just different, but totally erroneous.

Example

llama-mtmd-cli.exe -m LFM2-VL-1.6B-Q8_0.gguf --mmproj mmproj-LFM2-VL-1.6B-Q8_0.gguf -n 30 --image onefoul_512.png -p "OCR" -ngl 99 -t 0.0 -s 72

Outputs:

Guis mod est que motor modi modi doluplis deison et que que nosen-

That's the 512x64 image. Okay results so far. Now here's the 1024x64 image:

llama-mtmd-cli.exe -m LFM2-VL-1.6B-Q8_0.gguf --mmproj mmproj-LFM2-VL-1.6B-Q8_0.gguf -n 30 --image onefoul_1024.png -p "OCR." -ngl 99 -t 0.0 -s 72

Outputs:

Il semble que vous ayez une phrase ou un texte qui a été déformé ou distordu par une distorsion d'image ou une manipulation

Now if we use the original PyTorch model on the 1024x64 image, we get this output:

Cius modus est que modi dolipsis desion et que que nonsens

Much more relevant to the input image, unlike the GGUF.

Other weird behavior can be seen when running the model on images with 1024px on one side. I'm also currently in the process of fine-tuning OCR variants for LFM2-VL and noticed the unquantized GGUFs just giving broken responses for other images with 1024px.

It's as if the model isn't "seeing" the image properly anymore. If I understand correctly, LFM2-VL has some sort of tiled preprocessing and a global thumbnail for large images. Is all that logic fully implemented in llama.cpp for LFM2-VL?

Using GPU and/or disabling flash attention seems to make no difference. Images at 512 pixels or under have no issue.

I also tested on build b6195 - shortly after LFM2-VL was introduced - sadly similar behavior can be observed.

First Bad Commit

No response

Relevant log output

llama-mtmd-cli.exe -m LFM2-VL-1.6B-Q8_0.gguf --mmproj mmproj-LFM2-VL-1.6B-Q8_0.gguf -n 30 --image onefoul_1024.png -p "OCR." -ngl 99 -t 0.0 -s 72

...

clip_model_loader: model name:   LFM2 VL 1.6B
clip_model_loader: description:
clip_model_loader: GGUF version: 3
clip_model_loader: alignment:    32
clip_model_loader: n_tensors:    427
clip_model_loader: n_kv:         27

clip_model_loader: has vision encoder
clip_ctx: CLIP using CPU backend
load_hparams: projector:          lfm2
load_hparams: n_embd:             1152
load_hparams: n_head:             16
load_hparams: n_ff:               4304
load_hparams: n_layer:            26
load_hparams: ffn_op:             gelu
load_hparams: projection_dim:     2048

--- vision hparams ---
load_hparams: image_size:         256
load_hparams: patch_size:         16
load_hparams: has_llava_proj:     0
load_hparams: minicpmv_version:   0
load_hparams: n_merge:            2
load_hparams: n_wa_pattern:       0
load_hparams: image_min_pixels:   65536
load_hparams: image_max_pixels:   262144

load_hparams: model size:         537.96 MiB
load_hparams: metadata size:      0.15 MiB
alloc_compute_meta: warmup with image size = 512 x 512
alloc_compute_meta:        CPU compute buffer size =    30.31 MiB
alloc_compute_meta: graph splits = 1, nodes = 840
warmup: flash attention is enabled
main: loading model: LFM2-VL-1.6B-Q8_0.gguf
encoding image slice...
image slice encoded in 666 ms
decoding image batch 1/1, n_tokens_batch = 69
image decoded (batch 1/1) in 310 ms

"Guis mod est que motor modi modi doluplis deison et que que nosen-"


llama_perf_context_print:        load time =     318.87 ms
llama_perf_context_print: prompt eval time =    1076.26 ms /    80 tokens (   13.45 ms per token,    74.33 tokens per second)
llama_perf_context_print:        eval time =     935.66 ms /    24 runs   (   38.99 ms per token,    25.65 tokens per second)
llama_perf_context_print:       total time =    2403.07 ms /   104 tokens
llama_perf_context_print:    graphs reused =          0

llama-mtmd-cli.exe -m LFM2-VL-1.6B-Q8_0.gguf --mmproj mmproj-LFM2-VL-1.6B-Q8_0.gguf -n 30 --image onefoul_1024.png -p "OCR" -ngl 99 -t 0.0 -s 72

...

clip_model_loader: model name:   LFM2 VL 1.6B
clip_model_loader: description:
clip_model_loader: GGUF version: 3
clip_model_loader: alignment:    32
clip_model_loader: n_tensors:    427
clip_model_loader: n_kv:         27

clip_model_loader: has vision encoder
clip_ctx: CLIP using CPU backend
load_hparams: projector:          lfm2
load_hparams: n_embd:             1152
load_hparams: n_head:             16
load_hparams: n_ff:               4304
load_hparams: n_layer:            26
load_hparams: ffn_op:             gelu
load_hparams: projection_dim:     2048

--- vision hparams ---
load_hparams: image_size:         256
load_hparams: patch_size:         16
load_hparams: has_llava_proj:     0
load_hparams: minicpmv_version:   0
load_hparams: n_merge:            2
load_hparams: n_wa_pattern:       0
load_hparams: image_min_pixels:   65536
load_hparams: image_max_pixels:   262144

load_hparams: model size:         537.96 MiB
load_hparams: metadata size:      0.15 MiB
alloc_compute_meta: warmup with image size = 512 x 512
alloc_compute_meta:        CPU compute buffer size =    30.31 MiB
alloc_compute_meta: graph splits = 1, nodes = 840
warmup: flash attention is enabled
main: loading model: LFM2-VL-1.6B-Q8_0.gguf
encoding image slice...
image slice encoded in 621 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 346 ms

Il semble que vous ayez une phrase ou un texte qui a été inversé ou décalé. Pourriez-vous me donner plus de contexte

llama_perf_context_print:        load time =     316.52 ms
llama_perf_context_print: prompt eval time =    1047.88 ms /    75 tokens (   13.97 ms per token,    71.57 tokens per second)
llama_perf_context_print:        eval time =     929.30 ms /    29 runs   (   32.04 ms per token,    31.21 tokens per second)
llama_perf_context_print:       total time =    2395.44 ms /   104 tokens
llama_perf_context_print:    graphs reused =          0

[CISC]

Works flawlessly for me with BF16 LFM2-VL-450M on the images you provided at least (CPU and CUDA).

Is it exactly 1024px or >=1024px?

[SmartestWashingMachine]

Thanks for the quick response! It's tricky to pinpoint as sometimes the model still seems to give the correct (at least, equivalent to PyTorch) output from the looks of it.

I've noticed it > 1024px too.

Here's another one I just got with 450M FP16 if it helps:

512x31:

llama-mtmd-cli.exe -m LFM2-VL-450M-F16.gguf --mmproj mmproj-LFM2-VL-450M-F16.gguf -n 100 --image x1_512.png -p "What's the text in this image?" -ngl 99 -t 0.0 -s 72

GGUF:
"For the vision tower, LM2-VL uses SigmpVL, Nflax encoders to convert input images into token sequences. Two variants are implemented:"

PyTorch:
For the vision tower, LFM2-VL uses SigilLIP2 NaFlex encoders to convert input images into token sequences. Two variants are implemented:

Similar results. Now with 1100x67:

llama-mtmd-cli.exe -m LFM2-VL-450M-F16.gguf --mmproj mmproj-LFM2-VL-450M-F16.gguf -n 100 --image x1.png -p "What's the text in this image?" -ngl 99 -t 0.0 -s 72

GGUF:

"For the vision tower IEMV1, use Sigil ID N:EFLV and add the following code to convert input images into token sequences:

IEMV1:V1V1V1V1V1V1V1V1V1V1V1V1V1V1V1V1V1V1V1V1V1V1V1V1V1V1V1

(this would keep repeating)

PyTorch (which is also loaded in float16):
For the vision tower, LFM2-VL uses SigilLIP2 NAFlex encoders to convert input images into token sequences. Two variants are implemented:

EDIT:

This might be unrelated, but now I'm noticing (different?) divergent behavior on a certain 512x14 sample too for this 450M F16 variant:

llama-mtmd-cli.exe -m LFM2-VL-450M-F16.gguf --mmproj mmproj-LFM2-VL-450M-F16.gguf -n 100 --image trialtwo_512.png -p "What's the text in this image?" -ngl 99 -t 0.0 -s 72

GGUF:

The text in the image is as follows:

"...the ancient Greeks and just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just

PyTorch (float16):

This flexible architecture enables us to just the speed quality tradeoff during inference without

They're both F16 too.

EDIT 2:

Similar erroneous outputs on a WSL2 Ubuntu 22.04.3 LTS (CPU ver):

The 512x31 image reads (mostly correct):

The text in the image reads:

"For the vision tower, LM2-VL uses SigmpVL, Nflax encoders to convert input images into token sequences. Two variants are implemented:"

The 1100x67 reads:

The text in the image reads:

"For the vision tower IEMV1, use Sigil ID N:EFLV and add the following code to convert input images into token sequences:

IEMV1 uses Sigil ID N:EFLV and add the following code to convert input images into token sequences:

IEMV1 uses Sigil ID N:EFLV and add the following code to convert input images into token sequences:

IEMV1 uses

(repeats)

[CISC]

Right, these I can reproduce.

I recommend using BF16 instead of F16 BTW, just in case.

[SmartestWashingMachine]

Yeah, you're right. I don't like FP16 either. BF16 trials on my Windows machine:

Sample (1100x67)

llama-mtmd-cli.exe -m LFM2-VL-450M-BF16.gguf --mmproj mmproj-LFM2-VL-450M-BF16.gguf -n 100 --image x1.png -p "What's the text in this image?" -ngl 99 -t 0.0 -s 72

BF16 GGUF:

The text in the image reads:

"For the vision tower I.M.V.I, use Sigil ID N.ELF, add nodes to convert input images into token sequences. Two variants are implemented:"
"Two variants are implemented:"
"Two variants are implemented:"
"Two variants are implemented:"
"Two variants are implemented:"
"Two variants are implemented:"
"Two variants are implemented:"
"Two variants are implemented:"
"Two variants are implemented:"
"

(repeats)

BF16 Pytorch:

For the vision tower, LFM2-VL uses SigilLIP2 NAFlex encoders to convert input images into token sequences. Two variants are implemented:

Sample (512x31)

llama-mtmd-cli.exe -m LFM2-VL-450M-BF16.gguf --mmproj mmproj-LFM2-VL-450M-BF16.gguf -n 100 --image x1_512.png -p "What's the text in this image?" -ngl 99 -t 0.0 -s 72

BF16 GGUF:

The text in the image reads:

"For the vision tower, LM2-VL uses SigmpVL, Nflax encoders to convert input images into token sequences. Two variants are implemented:"

BF16 Pytorch:

For the vision tower, LFM2-VL uses SigilLIP2 NaFlex encoders to convert input images into token sequences. Two variants are implemented:

Sample (512x14)

llama-mtmd-cli.exe -m LFM2-VL-450M-BF16.gguf --mmproj mmproj-LFM2-VL-450M-BF16.gguf -n 100 --image 512x14.png -p "What's the text in this image?" -ngl 99 -t 0.0 -s 72

BF16 GGUF:

The text in the image is as follows:

"...the ancient Greeks and just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just just

(repeats)

BF16 PyTorch:

This flexible architecture enables us to just the speed quality tradeoff during inference without

[CISC]

Ok, this is a problem with the padding, if I manually pad the image to 512x512 it works perfectly.

[SmartestWashingMachine]

On the 512x14 sample, I'm also noticing if I pad the input image to 512x512 it's giving a good output. This also seems to be the case for 512x256 (at least for this sample).

512x512:

llama-mtmd-cli.exe -m LFM2-VL-450M-BF16.gguf --mmproj mmproj-LFM2-VL-450M-BF16.gguf -n 100 --image ja_test17.png -p "What's the text in this image?" -ngl 99 -t 0.0 -s 72

"The flexible architecture enables users to adapt the speed quality of the system during inference without..."

This appears to be a partial sentence or statement about a flexible architecture that allows users to adjust the speed and quality of a system during inference.

512x256:

llama-mtmd-cli.exe -m LFM2-VL-450M-BF16.gguf --mmproj mmproj-LFM2-VL-450M-BF16.gguf -n 100 --image ja_test18.png -p "What's the text in this image?" -ngl 99 -t 0.0 -s 72

"The flexible architecture enables users to adapt the speed of quality-tracking during inference without..."

This text appears to be a placeholder or a comment, as it is not complete and does not provide any specific information or context about the subject matter.

512x128:

llama-mtmd-cli.exe -m LFM2-VL-450M-BF16.gguf --mmproj mmproj-LFM2-VL-450M-BF16.gguf -n 100 --image ja_test19.png -p "What's the text in this image?" -ngl 99 -t 0.0 -s 72

The text in the image is:

"The best product ever made to go on the go. It was amazing."

512x255:

llama-mtmd-cli.exe -m LFM2-VL-450M-BF16.gguf --mmproj mmproj-LFM2-VL-450M-BF16.gguf -n 100 --image ja_test20.png -p "What's the text in this image?" -ngl 99 -t 0.0 -s 72

The text in the image is:

"The best answer is not the one that is asked. The best answer is not the one that is asked. The best answer is not the one that is asked. The best answer is not the one that is asked. The best answer is not the one that is asked. The best answer is not the one that is asked. The best answer is not the one that is asked. The best answer is not the one that is asked. The best answer is

(repeats)

EDIT (18 Nov)

At least for one long sample I'm trying (510x17)...

Is the input image expected to be padded at all in Pytorch? It's resized / "stretched" to the target dimensions (width=1408, height=64) from the Lfm2VlImageProcessorFast.resize_and_split -> images = F.resize(images, (new_height, new_width), interpolation=interpolation) looking like so:

(and then it's converted into tokens, padded to 1024, etc...)

If I disable the padding in the img_tool::resize call the image ends up looking like so:

Which is very similar to the one in Pytorch - only slight differences in visual appearance. Dimensions are the same. Unfortunately the outputs are still radically different, so this is probably a dead end too(?)

---

The only thing I've noticed so far in the llama-cpp pixel unshuffle implementation is that if (pad_width || pad_height) isn't called - pad_width and pad_height are 0 (at least for LFM2-VL on this sample).

From what I'm seeing in pixel_unshuffle shape results:

cur = ggml_reshape_3d(ctx0, cur, n_embd, width, height); -> (768 (n_embd), 88 (n_width_tokens), 4 (n_height_tokens), 1)

cur = ggml_reshape_3d(ctx0, cur, n_embd * scale_factor, width / scale_factor, height); -> (1536, 44, 4, 1)

cur = ggml_permute(ctx0, cur, 0, 2, 1, 3); -> (1536, 4, 44, 1)

cur = ggml_cont_3d(ctx0, cur, n_embd * scale_factor * scale_factor, height / scale_factor, width / scale_factor); -> (3072, 2, 44, 1)

cur = ggml_permute(ctx0, cur, 0, 2, 1, 3); -> (3072, 44, 2, 1)

cur = ggml_cont_2d(ctx0, cur, cur->ne[0], cur->ne[1] * cur->ne[2]); -> (3072, 88, 1, 1)

From what I'm seeing in resize_position_embeddings shape results:

pos_embd = ggml_reshape_3d(ctx0, pos_embd, n_embd, n_per_side, n_per_side); -> (768, 16, 16, 1)

pos_embd = ggml_permute(ctx0, pos_embd, 2, 0, 1, 3); -> (16, 16, 768, 1)

pos_embd = ggml_interpolate(ctx0, pos_embd, width, height, n_embd, 1, mode); -> (88, 4, 768, 1)

pos_embd = ggml_permute(ctx0, pos_embd, 1, 2, 0, 3); -> (768, 88, 4, 1)

pos_embd = ggml_cont_2d(ctx0, pos_embd, n_embd, width * height); -> (768, 352, 1, 1)

EDIT (20 Nov)

Even using a modified LFM2-VL-450M-BF16 where downsample_factor is set to 1 instead of 2 (so pixel unshuffle AKA build_patch_merge_permute has a scale_factor of 1) gives diverging results to the one in Pytorch. Tested with and without padding in img_tool::resize

Tiling probably isn't the issue here, as in Pytorch for this sample no tiling seems to be used - it's just one image being processed I think?

[CISC]

Think I found the problem, the scaling/padding goes a little nuts and unnecessarily upscales the image, then proceeds to erase most of the image instead of padding. I'll work on a fix.

[tdakhran]

@SmartestWashingMachine, the preprocessing in llama.cpp differs from that of PyTorch, as you noticed; it doesn't contain tiling.
@CISC the upscaling happens because of "smart resize", there is a min_image_tokens that the model was trained on, also width and height shall be divisible by const int total_factor = params.patch_size * params.proj_scale_factor;.

I used image_manipulation::resize_and_pad_image, but didn't look into the implementation.
Thanks for taking care of the fix. Happy to help if needed.

[CISC]

@SmartestWashingMachine, the preprocessing in llama.cpp differs from that of PyTorch, as you noticed; it doesn't contain tiling. @CISC the upscaling happens because of "smart resize", there is a min_image_tokens that the model was trained on, also width and height shall be divisible by const int total_factor = params.patch_size * params.proj_scale_factor;.

I used image_manipulation::resize_and_pad_image, but didn't look into the implementation. Thanks for taking care of the fix. Happy to help if needed.

Yeah, it's not specific to LFM2-VL, ideally upscaling should not be necessary when only one of the dimensions needs to be aligned, but the real issue is the erroneous padding when doing so.

Strike that, it seems the issue is a difference in padding, not sure exactly where it's handled, but it seems image is expected to be padded at the end instead of around all edges.

[CISC]

@tdakhran Sigh, been chasing ghosts, I think there's something wrong with the pos_embd resizing and/or pixel shuffle, but can't quite pin it down (other than that the -> (n_per_side, n_per_side, n_embd) comment does not match the permute).

Kimi-VL suffers from the same issue, while Qwen3-VL does not, suggesting it might be a width/height mixup.

[ngxson]

Btw @tdakhran I haven't yet looked into the pytorch impl, but do you also pad the image, or simply resize (stretching allowed)?

[tdakhran]

Thanks for looking into it @CISC.
@ngxson, no stretching, we keep the aspect ratio.

I wonder if it has anything to do with tiling implemented in PyTorch, or a different padding algorithm, will debug a bit next week.