Regenerating the models

[PriNova]

Now with the merged PR #252 I need to reconvert the model.
Does this mean, I need to reconvert the base models consolidated.0.pth or can I reconvert the quantized models too?

I'm currently working with the 7B alpaca Lora 4bit and the 13B alpaca lora model 4bit and would like to continue working with it.
I don't have the ressources to:

1. fine-tune the models again for using with llama.cpp
2. nor do I have the ressources to quantize the big models.

If I could reconvert my quantized alpaca lora models will be amazing.
If not, this would be a breaking change (as digital nomade often slow wifi and limited ressources)

It will be amazing, when someone could clarify.

Thank you

[j-f1]

You will need to start from the .pth and then convert to GGML, then quantize the resulting files again.

[PriNova]

Ok, this is then a breaking change for me, because I could not train for LoRa.

Instead of storing the tokenizer scores into the model, would it make more sense to store the scores in a seperate file?
Then the model can be used as is and has no direct dependency to the token vocab scores.

[Piezoid]

@eiz made a script to patch the ggml models by merging in the tokenizer with its weights: #324 (comment)

[PriNova]

Thank you, I found it and it works perfectly

[j-f1]

Note that we will be updating the model format soon to allow for instant loading and this will require regenerating from the .pth files again (although nothing will need to be retrained).

[PriNova]

Note that we will be updating the model format soon to allow for instant loading and this will require regenerating from the .pth files again (although nothing will need to be retrained).

Does I understand it correctly, that it will work with the pre-trained models also?

[PriNova]

Here are the both files which I issued in #324 for comparison.

tokens.txt
tokensSentencPiece.txt