3 Commits

Author	SHA1	Message	Date
goerch	afc4ca2889	convert : update convert-new.py with tokenizer fixes (#2614) ... * Merge tokenizer fixes into the gguf branch. * Add test vocabularies * Adapt convert-new.py (and fix a clang-cl compiler error on windows)	2023-08-14 20:20:04 +03:00
Georgi Gerganov	d2bb3ac10b	convert.py : remove GGML vocab + other obsolete stuff	2023-07-27 16:36:35 +03:00
Georgi Gerganov	68f53485e4	convert.py : start a new simplified implementation by removing old stuff	2023-07-27 15:56:53 +03:00