65B quantized for CPU

[satvikpendem]

Is there any way to run the 65B model on the CPU quantized for 4 bit? I saw that it's about 40 gigs for RAM usage when quantized.

How much RAM is required to quantize the 65B model? I'm not sure I have enough RAM to quantize myself, anyone have the model files for the quantized output for the 65B model for CPU? I've only found the quantized GPU files so far.

[jarcen]

Big models are split into chunks of 12 to 15 GB. Both pth->ggml converter and quantizer work with chunks so that at most one is loaded in memory. At each stage outputs are also chunks. If you can convert 13B model then I see nothing stopping you from doing same with 65B.

[G2G2G2G]

Yea it works the same as every other model and the same steps are in the readme...?

[satvikpendem]

Alright just tried it, it takes about 40 GB RAM to run. Thanks!