llama.cpp

Commit Graph

Author	SHA1	Message	Date
Anisse Astier	a808370c58	examples : new program to verify gguf tokenizer parameters This program verifies that a given gguf model file can tokenize all potential valid characters. Since llama.cpp currently raises an exception when tokenization is not possible[1], this tool helps verifying that valid ascii and utf-8 will always be properly tokenized. [1] https://github.com/ggerganov/llama.cpp/issues/2580	2024-04-30 08:46:04 +02:00

Author

SHA1

Message

Date

Anisse Astier

a808370c58

examples : new program to verify gguf tokenizer parameters

This program verifies that a given gguf model file can tokenize all
potential valid characters. Since llama.cpp currently raises an
exception when tokenization is not possible[1], this tool helps
verifying that valid ascii and utf-8 will always be properly tokenized.

[1] https://github.com/ggerganov/llama.cpp/issues/2580

2024-04-30 08:46:04 +02:00

1 Commits