Commit Graph

8 Commits

Author SHA1 Message Date
teleprint-me 78d7828cc4
chore: Add prototyped CLI options 2024-05-22 19:59:33 -04:00
teleprint-me 6fc4492b3f
chore: Add english pangram to vocab tests 2024-05-20 00:51:35 -04:00
teleprint-me a0362ea475
patch: Fix nested quotes for dict refs 2024-05-19 22:39:05 -04:00
teleprint-me 316b404d94
patch: Fix CLI option for generating vocab tests 2024-05-18 23:59:22 -04:00
teleprint-me da5deebda1
fix: Apply fix to verbose help description and generating vocab tests option 2024-05-18 23:34:33 -04:00
teleprint-me bd32266c87
feat: Add function for generating vocab script and fix CLI opts 2024-05-18 22:14:58 -04:00
teleprint-me 1a82573126
feat: Add example script for automating generating tokenizer model checksums and tests 2024-05-18 20:49:22 -04:00
teleprint-me 832b449cbd
feat: Add pre-tokenizer CLI tooling 2024-05-18 14:33:56 -04:00