Where is tokenizer.model?

[Thresher12]

Hello, sorry if this is a simple question but I am trying to convert the GPT4All model with the code giving in the description.

python3 convert-gpt4all-to-ggml.py models/gpt4all-7B/gpt4all-lora-quantized.bin ./models/tokenizer.model

but there is no such tokenizer.model file in the repo, no hint on where to get it and even googling comes up with nothing. Where are you supposed to get this file? thanks

[Thresher12]

I randomly found a suitable tokenizer.model among the gpt4-x-alpaca files

[arkilis]

@Thresher12 Facing the same issue, is it possible to share that file pls? Thank you

[Thresher12]

@Thresher12 Facing the same issue, is it possible to share that file pls? Thank you

Its here. It would be nice to have links or some hint as to where to find to all the required dependencies that aren't already included.

https://huggingface.co/chavinlo/gpt4-x-alpaca/tree/main