3 Commits

Author	SHA1	Message	Date
Georgi Gerganov	123175ea71	common : gpt_params_parse do not print usage	2024-06-03 14:51:31 +03:00
Jared Van Bortel	1b67731e18	BERT tokenizer fixes (#6498) ... Key changes: * BERT conversion: fix abuse of LlamaHfVocab, do not set BOS or EOS * Nomic Embed conversion: pad vocab instead of slicing embedding tensor * llama_tokenize: handle added special tokens like HF does	2024-04-09 13:44:08 -04:00
Johannes Gäßler	50ccaf5eac	lookup: complement data from context with general text statistics (#5479) ... * lookup: evaluation tools, use corpus/previous gens * fixup! lookup: evaluation tools, use corpus/previous gens * fixup! lookup: evaluation tools, use corpus/previous gens * fixup! lookup: evaluation tools, use corpus/previous gens * fixup! lookup: evaluation tools, use corpus/previous gens	2024-03-23 01:24:36 +01:00