Mixtral 8x7b models using more memory while loading

[RyenNelsen]

There appears to be a regression between release versions b2586 and b2589. When attempting to load Mixtral 8x7b models with any version greater than b2586, the system utilizes an abnormal amount of memory compared to previous versions. Manually disabling mmap does resolve the issue.

Platform:
Windows 11 Pro
64GB RAM
Nvidia 3080

Example command:
.\main.exe -m 'C:\models\dolphin-2.7-mixtral-8x7b.Q5_0.gguf' -p "<|im_start|>user\nHello!\n<|im_end|>\n<|im_start|>assistant\n"

Versions I tested:
b2586 cuda cu12.2.0 & openblas

b2589 cuda cu12.2.0 & openblas

b2589 avx512

Diffing log output from b2586 cuda cu12.2.0 and b2589 cuda cu12.2.0 shows the following:
b2586: llm_load_tensors: CPU buffer size = 30735.50 MiB
b2589: llm_load_tensors: CUDA_Host buffer size = 30735.50 MiB

[compilade]

This may be related to #6387. If I'm understanding this correctly, a solution would be to re-convert the model to GGUF from the original model files. (@slaren might want to clarify)

I didn't yet find a recent re-conversion of dolphin-mixtral-8x7b on HuggingFace but someone might do it eventually.
But there are already finetunes which use the new format, like https://huggingface.co/bartowski/Tess-2.0-Mixtral-v0.2-GGUF.

[LostRuins]

I can confirm this seems to happen for me too, I'm getting OOM at configs which worked fine previously.

[phymbert]

Yes that's why #6387 is a breaking changes. You need to convert to GGUF again to have merged experts tensors per layer or disable mmap.

It is clearly stated here:
#6387 (comment)

[aleksusklim]

@slaren, @phymbert, can you look please?
LostRuins#786