[fixed]The last code build with memory fix running result is not good in my pc.

[FNsi]

Be obviously slower with Q_1 30b model. And the memory usage become garbage...
(Linux 5.19 x64 Ubuntu base)

[FNsi]

Seems the flag n makes different.
n=100000 no long work🤔.
But After change n to 4096 also not working...

[FNsi]

Seems the flag n makes different.
n=100000 no long work🤔.
After change n to 4096 also not working.

Rollback to my back up...

[Green-Sky]

some of the last commits changed/touched how memory is handled.
also there is -c you can set up to 2048.

[x02Sylvie]

Kinda similar case here, although im unsure which specific commit caused performance loss

After swapping out old exe for new one, I went from 207 ms per token to 269 on 13b alpaca
I suspect impact might be more noticable on 30b and 65b models in this case

[FNsi]

some of the last commits changed/touched how memory is handled.

also there is -c you can set up to 2048.

Always. -c works fine even with 5000+ in my pc😅 so I guess there be somewhere else problems.

[FNsi]

Kinda similar case here, although im unsure which specific commit caused performance loss

After swapping out old exe for new one, I went from 207 ms per token to 269 on 13b alpaca

I suspect impact might be more noticable on 30b and 65b models in this case

Yes. In 30b I was enable to chat with more than half an hour, now with less than 20 tokens the ggml show memory not enough.

[Green-Sky]

@FNsi please try again with latest master.

[FNsi]

@FNsi please try again with latest master.

Ban the Blas make it works, still have little performance loss anyway.

a guess from me, is it because the blas try to make 4 bit prompt back to 16 bit in run time...? 😅😂

[ggerganov]

It's not the 4-bits - it does not work with F16 either.
I am almost sure that this F16 BLAS call is somehow wrong (as well as the rest of them):

https://github.com/ggerganov/llama.cpp/blob/8520fc310eab87f2c4612f2a00d4adbd44a20d0d/ggml.c#L6244-L6250

Which is super strange since this has been used in whisper.cpp forever and it seems to work..

[FNsi]

It's not the 4-bits - it does not work with F16 either.

I am almost sure that this F16 BLAS call is somehow wrong:

https://github.com/ggerganov/llama.cpp/blob/8520fc310eab87f2c4612f2a00d4adbd44a20d0d/ggml.c#L6244-L6250

Which is super strange since this has been used in whisper.cpp forever and it seems to work..

So, it seems mean that, a big performance improvement is coming as if it's been figured out.