Commit Graph

6 Commits

Author SHA1 Message Date
jaime-m-p 77cbb79532 Refactor random tokenizer test 2024-05-08 23:30:52 +02:00
jaime-m-p 7761f8ea71 Ignore special tokens for testing 2024-05-08 00:45:16 +02:00
jaime-m-p a5fa2fec60 Style 2024-05-07 20:38:00 +02:00
jaime-m-p 67832e5554 llama3 custom regex split: fix \s 2024-05-05 01:20:23 +02:00
jaime-m-p 69a49ac3a1 Fix merge 2024-05-05 00:42:44 +02:00
jaime-m-p 3e3e2838a1 Add bruteforce random tests for token encoding 2024-05-04 01:34:36 +02:00