Unicode support

[wizd]

Thannk you for creating such a great inference engine which has 10x speedup.
Please add Unocode support to display other language properly.

[beiller]

I tried to determine how to implement unicode and I am not getting far. It seems to work from all I am seeing, but the output has random characters yes.

Here is a prompt in text format for easier copy/paste

./main -m ./models/13B/ggml-model-q4_0.bin -t 8 -n 128 -n 512 -p $'人生の意味は'

     1 -> ''
 30313 -> '人'
 30486 -> '生'
 30199 -> 'の'
 31474 -> '意'

This seems correct above since I dumped out the tokens parsing code

llama_model_load: vocab[30313] = '人'
llama_model_load: vocab[30486] = '生'
llama_model_load: vocab[30199] = 'の'
llama_model_load: vocab[31474] = '意'

And the output I get is

人生の意���、フロントカードに���いてる。 2019年3月　© All Rights Reserved. [end of text]

So it is outputting some characters but some �

llama_model_load: vocab[30140] = '�'

[beiller]

I find a list of unprintable tokens from ID 131 to 258. If I remove those from vocab a prompt can generate in Japanese it seems but I dont know Japanese!

llama.cpp % ./main -m ./models/13B/ggml-model-q4_0.bin -t 8 -n 128 --repeat_last_n 64 --repeat_penalty 1.0 -n 512 -p $'人生の意味は'

Response

人生の意は、一人が一人ということであります。は安部が立していたので、去からは一人の人にれるのはにとどまったのですが、そう

Google translate

The meaning of life is that one person is one person. Since Abe was standing there, it was only possible to be one person after leaving, but that's right.

Is it possible?

[blackhole89]

Response

人生の意は、一人が一人ということであります。は安部が立していたので、去からは一人の人にれるのはにとどまったのですが、そう

The Japanese text you quote here is fairly agrammatical, in a way that suggests (on top of some other issues that I figure are simply due to LLaMa not having learned the language very well) that some words are simply missing. Where were the unprintable tokens that you removed from this?

[beiller]

I removed "", "�", "��" from the grammar, not from a sentence that's not how it works. There is a large chunk of the "token dictionary" in the model that points to unprintable character �. I remove those tokens from the dictionary of tokens the program is using. I suspect the model learns some corrupted text maybe during training so if it sees japanese characters it is confusing it with some garbled text it has come across, thus making unprintable characters a likely candidate for the next word. Just my hypothesis.

Here is the pull request, the code change I made to make this work.

https://github.com/ggerganov/llama.cpp/pull/26/files

[beiller]

For anyone interested here is the chunk in the 13B model file. Not sure if all models contain the same token grammars

� vocab[131] (EFBFBD)
� vocab[132] (EFBFBD)
� vocab[133] (EFBFBD)
� vocab[134] (EFBFBD)
� vocab[135] (EFBFBD)
� vocab[136] (EFBFBD)
� vocab[137] (EFBFBD)
� vocab[138] (EFBFBD)
� vocab[139] (EFBFBD)
� vocab[140] (EFBFBD)
� vocab[141] (EFBFBD)
� vocab[142] (EFBFBD)
� vocab[143] (EFBFBD)
� vocab[144] (EFBFBD)
� vocab[145] (EFBFBD)
� vocab[146] (EFBFBD)
� vocab[147] (EFBFBD)
� vocab[148] (EFBFBD)
� vocab[149] (EFBFBD)
� vocab[150] (EFBFBD)
� vocab[151] (EFBFBD)
� vocab[152] (EFBFBD)
� vocab[153] (EFBFBD)
� vocab[154] (EFBFBD)
� vocab[155] (EFBFBD)
� vocab[156] (EFBFBD)
� vocab[157] (EFBFBD)
� vocab[158] (EFBFBD)
� vocab[159] (EFBFBD)
� vocab[160] (EFBFBD)
� vocab[161] (EFBFBD)
� vocab[162] (EFBFBD)
� vocab[163] (EFBFBD)
� vocab[164] (EFBFBD)
� vocab[165] (EFBFBD)
� vocab[166] (EFBFBD)
� vocab[167] (EFBFBD)
� vocab[168] (EFBFBD)
� vocab[169] (EFBFBD)
� vocab[170] (EFBFBD)
� vocab[171] (EFBFBD)
� vocab[172] (EFBFBD)
� vocab[173] (EFBFBD)
� vocab[174] (EFBFBD)
� vocab[175] (EFBFBD)
� vocab[176] (EFBFBD)
� vocab[177] (EFBFBD)
� vocab[178] (EFBFBD)
� vocab[179] (EFBFBD)
� vocab[180] (EFBFBD)
� vocab[181] (EFBFBD)
� vocab[182] (EFBFBD)
� vocab[183] (EFBFBD)
� vocab[184] (EFBFBD)
� vocab[185] (EFBFBD)
� vocab[186] (EFBFBD)
� vocab[187] (EFBFBD)
� vocab[188] (EFBFBD)
� vocab[189] (EFBFBD)
� vocab[190] (EFBFBD)
� vocab[191] (EFBFBD)
� vocab[192] (EFBFBD)
� vocab[193] (EFBFBD)
� vocab[194] (EFBFBD)
� vocab[195] (EFBFBD)
� vocab[196] (EFBFBD)
� vocab[197] (EFBFBD)
� vocab[198] (EFBFBD)
� vocab[199] (EFBFBD)
� vocab[200] (EFBFBD)
� vocab[201] (EFBFBD)
� vocab[202] (EFBFBD)
� vocab[203] (EFBFBD)
� vocab[204] (EFBFBD)
� vocab[205] (EFBFBD)
� vocab[206] (EFBFBD)
� vocab[207] (EFBFBD)
� vocab[208] (EFBFBD)
� vocab[209] (EFBFBD)
� vocab[210] (EFBFBD)
� vocab[211] (EFBFBD)
� vocab[212] (EFBFBD)
� vocab[213] (EFBFBD)
� vocab[214] (EFBFBD)
� vocab[215] (EFBFBD)
� vocab[216] (EFBFBD)
� vocab[217] (EFBFBD)
� vocab[218] (EFBFBD)
� vocab[219] (EFBFBD)
� vocab[220] (EFBFBD)
� vocab[221] (EFBFBD)
� vocab[222] (EFBFBD)
� vocab[223] (EFBFBD)
� vocab[224] (EFBFBD)
� vocab[225] (EFBFBD)
� vocab[226] (EFBFBD)
� vocab[227] (EFBFBD)
� vocab[228] (EFBFBD)
� vocab[229] (EFBFBD)
� vocab[230] (EFBFBD)
� vocab[231] (EFBFBD)
� vocab[232] (EFBFBD)
� vocab[233] (EFBFBD)
� vocab[234] (EFBFBD)
� vocab[235] (EFBFBD)
� vocab[236] (EFBFBD)
� vocab[237] (EFBFBD)
� vocab[238] (EFBFBD)
� vocab[239] (EFBFBD)
� vocab[240] (EFBFBD)
� vocab[241] (EFBFBD)
� vocab[242] (EFBFBD)
� vocab[243] (EFBFBD)
� vocab[244] (EFBFBD)
� vocab[245] (EFBFBD)
� vocab[246] (EFBFBD)
� vocab[247] (EFBFBD)
� vocab[248] (EFBFBD)
� vocab[249] (EFBFBD)
� vocab[250] (EFBFBD)
� vocab[251] (EFBFBD)
� vocab[252] (EFBFBD)
� vocab[253] (EFBFBD)
� vocab[254] (EFBFBD)
� vocab[255] (EFBFBD)
� vocab[256] (EFBFBD)
� vocab[257] (EFBFBD)
� vocab[258] (EFBFBD)
�� vocab[26308] (EFBFBDEFBFBD)
 vocab[31634] (EFBFBC)

Many token IDs point to 0xEFBFBD which is unprintable unicode

[wizd]

Nice find!

Due to the constently changing encoding history of CJK (Chinese, Japanese, Korean), there is big chance that the training model got wrong encoding of non-ascii language. Simply removing it is good.

[wizd]

Some more test shows that we can't simply remove the unprintable token. There should be some way to find the right encoding of it. otherwise the generated text becomes unreadable.

[beiller]

Im not sure you have applied the code change. I cannot try your prompt since its an image mind pasting? But I think you have to checkout from my repo because my code is not currently merged here yet.

https://github.com/beiller/llama.cpp

Try clone / build in a different folder. It also includes the repeat penalty change. Again my approach is not about removing the characters, its a code change that will output something very different (and more accurate)

[beiller]

Heres some more examples:

./main -m ./models/13B/ggml-model-q4_0.bin -t 8 -p $'人生の意味は'

Outputs:

...
main: prompt: '人生の意味は'
main: number of tokens in prompt = 5
     1 -> ''
 30313 -> '人'
 30486 -> '生'
 30199 -> 'の'
 31474 -> '意'

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000


人生の意はあまり
物学に世界をする発見 : 空気中でニトルを放つシステムが全球ワー... [end of text]

人生の意から
しようごいます。
I’ve been getting these error messages on my blog for a while now, but I didn't pay too much attention to them until recently when there were quite few. Maybe it was because last weekend (in Japan), the phone lines at my company went down and that screwed up all of our accounts that require login through Internet Explorer 7+ (which is set as default).
Unfortunately, I couldn't afford much time to fix them since they were so many. So now there are even more errors for you guys

人生の意とか、やりめているんだ [ 1 ]
部事情はもうそこまでのキャバラ
く立みなしに上下。自分がよききたかったと、子あればえるだけ作らない人は多数だ [ 2 ]
【キャバラ】 (ビジネスの了)
く

人生の意は知らない。 我が人生は事の意や子をう きた人の自分である人生に存するから よく、意のい期に実力から人の意を心したり、知らない人生の意は子をうしていることが、