What makes this version so lighter and faster ?

[hijcard]

Hello,
first of all, thank you very much for this authentic piece of GOLD!
I just wanted to understand why the official version from facebook, as well as alpaca and vicuna, require a lot RAM to run on the CPU or a lot of memory to run on the GPU, while this version can successfully be used on my CPU with only 16GB RAM decently "fast".
Thanks

[omarkazmi]

[hijcard]

Yea dude, thanks for stating the obvious. I was looking for a technical explanation though

[jarcen]

This repo supports 4 bit quantization. Models released by researchers are usually in full or half precision. Those who have supercomputers are not pressured to squeeze models into Raspberry Pi and 4 bit quantization doesn't work well with training anyways.

Different priorities, simply speaking. Researchers are less interested in delivering working product.