Commit Graph

3 Commits

Author SHA1 Message Date
Georgi Gerganov 43e12ce8e5
llama : use new pre-tokenizer type 2024-04-26 20:08:57 +03:00
Georgi Gerganov e1b2bf783e
tests : add sample usage 2024-04-26 13:43:54 +03:00
Georgi Gerganov aeafb43ed7
tests : remove and rename tokenizer test scripts 2024-04-26 13:39:03 +03:00
Renamed from tests/test-tokenizer-0-llama.py (Browse further)