happyz
/
llama.cpp
mirror of https://github.com/ggerganov/llama.cpp.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
llama.cpp/examples/tokenizer-verifier
History
Anisse Astier a808370c58 examples : new program to verify gguf tokenizer parameters 
This program verifies that a given gguf model file can tokenize all
potential valid characters. Since llama.cpp currently raises an
exception when tokenization is not possible[1], this tool helps
verifying that valid ascii and utf-8 will always be properly tokenized.

[1] https://github.com/ggerganov/llama.cpp/issues/2580
 		2024-04-30 08:46:04 +02:00
..
CMakeLists.txt examples : new program to verify gguf tokenizer parameters 2024-04-30 08:46:04 +02:00
tokenizer-verifier.cpp examples : new program to verify gguf tokenizer parameters 2024-04-30 08:46:04 +02:00