happyz
/
llama.cpp
mirror of https://github.com/ggerganov/llama.cpp.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,784 Commits 567 Branches 6,117 Tags 1.7 GiB
Commit Graph

2 Commits

Author SHA1 Message Date
Georgi Gerganov d53240ccc2
refact : add tokenizer model 2024-05-03 17:27:12 +03:00
Galunid daab3d7f45
Add more tokenizer tests (#3742) 
* Add more tokenizer tests

* Add starcoder

* Update test vocab files

* Restrict bpe tokenizer tests to unicode planes

* Update comment

* Comment cosmetics

* Remove bloom vocab/test
 2023-10-24 09:17:17 +02:00