49c25cce19
* Merge tokenizer fixes into the gguf branch. * Add test vocabularies * Adapt convert-new.py (and fix a clang-cl compiler error on windows) * Improved tokenizer test But does it work on MacOS? * Improve token type support - Added @klosax code to convert.py - Improved token type support in vocabulary * Exclude platform dependent tests * More sentencepiece compatibility by eliminating magic numbers * Restored accidentally removed comment * Improve commentary * Use token type API in test-tokenizer-1.cpp |
||
|---|---|---|
| .. | ||
| CMakeLists.txt | ||
| test-double-float.cpp | ||
| test-grad0.cpp | ||
| test-grammar-parser.cpp | ||
| test-llama-grammar.cpp | ||
| test-opt.cpp | ||
| test-quantize-fns.cpp | ||
| test-quantize-perf.cpp | ||
| test-sampling.cpp | ||
| test-tokenizer-0.cpp | ||
| test-tokenizer-1.cpp | ||