3 Commits

Author	SHA1	Message	Date
jaime-m-p	d67de1a364	Merge commit '148995e5' into tokenizer-bpe-fixes	2024-06-12 00:24:20 +02:00
Jared Van Bortel	1b67731e18	BERT tokenizer fixes (#6498) ... Key changes: * BERT conversion: fix abuse of LlamaHfVocab, do not set BOS or EOS * Nomic Embed conversion: pad vocab instead of slicing embedding tensor * llama_tokenize: handle added special tokens like HF does	2024-04-09 13:44:08 -04:00
Johannes Gäßler	50ccaf5eac	lookup: complement data from context with general text statistics (#5479) ... * lookup: evaluation tools, use corpus/previous gens * fixup! lookup: evaluation tools, use corpus/previous gens * fixup! lookup: evaluation tools, use corpus/previous gens * fixup! lookup: evaluation tools, use corpus/previous gens * fixup! lookup: evaluation tools, use corpus/previous gens	2024-03-23 01:24:36 +01:00