0479e9695f
patch: Add exception handling for non-existent vocab related files
2024-05-18 22:14:19 -04:00
006bb60d27
chore: Fix model path references
2024-05-18 19:20:19 -04:00
2ef73ee6e4
refactor: Apply SoC for HF requests, vocab, and weights
2024-05-18 13:45:21 -04:00
5eda2c9485
feat: Add pre-tokenizer logging
2024-05-18 13:21:22 -04:00
b2ca23c746
feat: Add method for generating the checksums and writing the results to a json file
2024-05-18 01:46:13 -04:00
302258721b
refactor: Apply model schema to tokenizer downloads
...
- Add imports for json and hashlib
- Add missing models: phi, stablelm, mistral, and mixtral
- Fix constructor logic
- Fix how models are accessed
- Apply model schema to download_model method
2024-05-18 01:26:39 -04:00
f7515abf49
feat: Add tokenizer types, model types, and model repos
2024-05-18 00:37:19 -04:00
1a286c8e21
refactor: Clean up variable names and separate concerns when downloading tokenizers
2024-05-17 23:27:30 -04:00
5c8144e645
feat: Add download_model method and fix references for clarity to mitigate confusion
2024-05-17 23:00:12 -04:00
4790f76740
feat: Add prototype for requesting vocab related files
2024-05-17 21:08:39 -04:00
98cf788990
patch: Apply minor fixes for handling headers and writing content
2024-05-17 21:07:51 -04:00
742abebb39
refactor: Add log for status and fix url path variable name
2024-05-17 20:37:59 -04:00
ba13d64bb3
feat: Add utils for logging and writing when interacting with HuggingFaceHub
2024-05-17 20:26:21 -04:00
dbdf6c2b1d
feat: Add prototype for managing huggingface hub content
2024-05-17 20:00:48 -04:00
teleprint-me