b3a54291cb
Merge branch 'huggingface-hub-api' into auto-model-support
2024-05-25 20:28:40 -04:00
e4275bcef4
feat: Add example script for downloading models
2024-05-25 19:12:34 -04:00
fcd20ab9e9
chore: Add comments for each file extension type
2024-05-25 19:12:16 -04:00
da72554f58
feat: Add static methods for resolving model types and model extensions
2024-05-25 19:11:56 -04:00
f30bd63252
refactor: Add function for building and parsing CLI arguments
2024-05-25 14:41:13 -04:00
e9759dee0b
docs: Add revisions to hub-vocab.py module level docstring
2024-05-25 14:33:23 -04:00
6c1b0111a1
refactor: Apply huggingface_hub api to CLI
2024-05-25 04:16:10 -04:00
63c3410492
refactor: Add support for model file types
2024-05-25 04:15:39 -04:00
2ffe6b89c8
Refactor HFubModel and HFHubTokenizer to fix reference issues
2024-05-25 04:15:15 -04:00
fda2319d7b
refactor: Streamline method signatures and clarify method names related to downloading repo files
2024-05-25 03:32:27 -04:00
4438d052aa
refactor: Abstract file and logger management to streamline api interface
2024-05-25 02:57:59 -04:00
99275a1606
refactor: Simplify API and merge HFModel into HFHub
2024-05-25 02:10:52 -04:00
168297f11c
refactor: Add remote repository listings to the bas HFHub class
2024-05-24 23:57:45 -04:00
6da2bd6fbc
patch: Apply fix for paths and logging
2024-05-24 21:47:47 -04:00
64096942ce
refactor: Simplify the huggingface hub api to enable flexible model requests
2024-05-24 02:40:34 -04:00
6c9ac0fc52
refactor: Add a custom tokenizer component and fix vocab request class
2024-05-24 01:30:29 -04:00
e62e09bbb1
refactor: Apply fix for file path references
2024-05-23 22:59:16 -04:00
c91dcdf2a4
refactor: Add fixes for logging
2024-05-23 22:58:03 -04:00
77bc7394c8
refactor: Add tokenizer path, add methods for extracting vocab metadata, fix checksum method name
2024-05-23 21:40:05 -04:00
b4b553fe6c
chore: Apply ruff formatting for readability
2024-05-23 21:36:51 -04:00
ea4fc1095e
refactor: Apply fixes to required arguments and fixes to options
2024-05-23 21:36:31 -04:00
f62080adfa
refactor: Simplify huggingface hub vocab request
2024-05-23 20:50:58 -04:00
1749209406
refactor: Simplify huggingface hub api implementation
2024-05-23 20:50:15 -04:00
c92c6ad480
feat: Add CLI tool for fetching vocab files
2024-05-23 20:33:12 -04:00
0ccf579242
refactor: Apply consistent naming conventions
2024-05-23 17:17:22 -04:00
9ba6b92c2d
chore: Add required vocabulary constants
2024-05-23 16:57:14 -04:00
9814b7f9ab
feat: Add custom huggingface hub api
...
Signed-off-by: teleprint-me <77757836+teleprint-me@users.noreply.github.com>
2024-05-23 13:48:20 -04:00
e84b71c2c6
ggml : drop support for QK_K=64 ( #7473 )
...
* ggml : drop support for QK_K=64
ggml-ci
* opencl : restore QK_K=256 define
2024-05-23 10:00:21 +03:00
78d7828cc4
chore: Add prototyped CLI options
2024-05-22 19:59:33 -04:00
cd00be886f
chore: Add model metadata
2024-05-22 19:59:13 -04:00
1957ca41f2
refactor: Simplify BPE pre-tokenizer mapping
2024-05-22 16:57:29 -04:00
12285b5325
chore: Map model file and vocab types
2024-05-22 02:58:12 -04:00
0b43e14030
refactor: Add experimental mapping for BPE pre-tokenizers
2024-05-21 22:45:45 -04:00
34e14ae96d
refactor: Add experimental model mappings
2024-05-21 19:11:51 -04:00
201cc11afa
llama : add phi3 128K model support ( #7225 )
...
* add phi3 128k support in convert-hf-to-gguf
* add phi3 128k support in cuda
* address build warnings on llama.cpp
* adjust index value in cuda long rope freq factors
* add long rope support in ggml cpu backend
* make freq factors only depend on ctx size
* remove unused rope scaling type 'su' frin gguf converter
* fix flint warnings on convert-hf-to-gguf.py
* set to the short freq factor when context size is small than trained context size
* add one line of comments
* metal : support rope freq_factors
* ggml : update ggml_rope_ext API to support freq. factors
* backends : add dev messages to support rope freq. factors
* minor : style
* tests : update to use new rope API
* backends : fix pragma semicolons
* minor : cleanup
* llama : move rope factors from KV header to tensors
* llama : remove tmp assert
* cuda : fix compile warning
* convert : read/write n_head_kv
* llama : fix uninitialized tensors
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2024-05-21 23:28:32 +03:00
b2aac685d5
docs: Fix comment
2024-05-21 16:07:12 -04:00
83b9fcd3e4
refactor: Rename constants to reduce confusion between references
2024-05-21 16:06:39 -04:00
2fe28ad4d3
chore: Rename from repo to model repo and reorder for improved readability
2024-05-21 01:41:35 -04:00
4768650aff
chore: Add formatting, set common vocab files, apply pattern to model map
2024-05-21 01:38:29 -04:00
fb32f50834
feat: Add hf model mapping descriptors for each repo
2024-05-21 01:07:13 -04:00
a35b76755f
Merge branch 'master' into auto-model-support
2024-05-21 00:16:34 -04:00
aed0573f68
proto: Add experimental vocab pre-tokenizer regular expressions
2024-05-21 00:14:26 -04:00
5978bb007d
chore: Fix and update comments
2024-05-20 14:59:40 -04:00
2fa2c7a86c
chore: Move enums and model map to constants
2024-05-20 14:51:03 -04:00
d9ba963cd4
refactor: Restructure tokenizer model metadata
2024-05-20 14:42:59 -04:00
18bb36e496
chore: Allow the user to config the logger
2024-05-20 14:06:21 -04:00
fabf30b4c4
llama : remove Persimmon ( #7408 )
...
* llama : remove Persimmon
* requirements : remove
2024-05-21 02:35:28 +10:00
bdd0286bd0
refactor: Use proper names for referenced member variables
2024-05-20 01:39:09 -04:00
a1951e27dc
refactor: Add proper names for remote model references
2024-05-20 01:36:44 -04:00
6fc4492b3f
chore: Add english pangram to vocab tests
2024-05-20 00:51:35 -04:00
teleprint-me
Georgi Gerganov
liuwei-git