228b1bd487
Update README.md
2024-05-27 09:49:55 -07:00
1d6d9497a8
readme
2024-05-27 12:36:57 -04:00
de26d49fbe
duo: v5
2024-05-25 22:19:23 -04:00
7c8699add6
pass user data
2024-05-25 22:10:19 -04:00
534093878b
duo: v3
2024-05-25 14:41:30 -04:00
96811fdf63
duo: v2
2024-05-25 14:23:57 -04:00
78938bc0c9
duo: v0
2024-05-25 13:59:28 -04:00
83aabb3fb7
readme
2024-05-24 23:56:48 -04:00
10d5aefed5
logging
2024-05-24 22:21:41 -04:00
66982abcb1
fixes
2024-05-24 12:22:59 -04:00
02e2c91d01
correct split id
2024-05-24 09:52:28 -04:00
60fe62e6eb
some renaming
2024-05-22 23:52:36 -04:00
479c80a0db
duo: cleanup v2
2024-05-22 23:31:23 -04:00
eecdd3b0ce
duo: first ~working option
2024-05-22 23:02:31 -04:00
2849247c4f
duo: more cleanup
2024-05-21 22:45:59 -04:00
f3965704fd
duo: simplify a little
2024-05-21 22:31:52 -04:00
d52d193e58
duo v0
...
setting up RPC + callback on each split completion
1. start rpc server on local instance on two different ports with 5GB
allocated each.
2. set up another callback on completion of a split. This seems cleaner
than trying to second-guess which tensor is the boundary of a split.
3. run it with 8B model @ 4bit, observe split_done captured at a reasonable place.
Next step - bring back linear speculation and start speculating on another remote
instances.
2024-05-21 16:11:30 -04:00
11474e756d
examples: cache hf model when --model not provided ( #7353 )
...
* examples: cache hf model when --model not provided
* examples: cache hf model when --model not provided
* examples: cache hf model when --model not provided
* examples: cache hf model when --model not provided
* examples: cache hf model when --model not provided
2024-05-21 17:13:12 +03:00
d7e852c1bc
Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) ( #7425 )
...
* Update brute force test: add_special
* Update brute force test: default values for add_bos_token and add_eos_token
* Enable rtrim when pre-inserting BOS
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Revert "server : fix test regexes"
2024-05-21 14:39:48 +02:00
917dc8cfa6
Tokenizer SPM fixes for phi-3 and llama-spm ( #7375 )
...
* Update brute force test: special tokens
* Fix added tokens
- Try to read 'added_tokens.json'.
- Try to read 'tokenizer_config.json'.
- Try to read 'tokenizer.json'.
* Fix special tokens rtrim
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* server : fix test regexes
2024-05-20 20:15:57 +02:00
20385cebcc
perplexity: update README FP16 results [no ci] ( #7413 )
2024-05-20 18:15:38 +02:00
3bc10cb485
server : fix temperature + disable some tests ( #7409 )
...
* server : fix temperature
* server : disable tests relying on parallel determinism
* ci : change server Debug -> RelWithDebInfo
2024-05-20 22:10:03 +10:00
1cc0155d04
server : tuning tests ( #7388 )
...
* server : don't pass temperature as string
* server : increase timeout
* tests : fix the fix 0.8f -> 0.8
ggml-ci
* tests : set explicit temperature
2024-05-20 10:16:41 +03:00
e932094d58
server : return error on too large embedding input ( #7389 )
2024-05-20 08:56:05 +03:00
2789baf480
tests : fix --keep_split -> --keep-split ( #7374 )
2024-05-20 08:55:09 +03:00
1ea2a0036e
quantize : fix --keep-split check ( #7374 )
2024-05-19 19:37:04 +03:00
1b01f06db0
server: add test for token probs ( #7347 )
2024-05-19 16:26:02 +02:00
41858392e1
server: fix seed being reported back ( #7382 )
2024-05-19 17:06:33 +03:00
854d365aba
cmake : update android comments ( #7341 )
2024-05-19 11:01:01 +03:00
511182eabb
android : use "ci-android" branch for CI ( #7341 )
...
* android : use "ci-android" branch for CI
* ggml : disable SIMD exp and silu for 32-bit ARM
ggml-ci
* android : do not fetch, use add_subdirectory instead
* cmake : provide binary dir
2024-05-18 20:40:39 +10:00
cb42c29427
server: correct --threads documentation [no ci] ( #7362 )
2024-05-18 11:10:47 +02:00
ca57e0f35e
perplexity : ndot progress and show stats with < 100 tasks ( #7348 )
...
Fix floating point error with ndot printing, allow end stats on lower task numbers if multiple-choice tasks.
2024-05-18 10:57:08 +03:00
f4bd8b3d26
rpc : set SO_REUSEADDR for the server socket ( #7320 )
...
ref: #7293
2024-05-17 17:25:44 +03:00
ee94172d33
server : add support for the RPC backend ( #7305 )
...
ref: #7292
2024-05-17 10:00:17 +03:00
9c4fdcbec8
[Server] Added --verbose option to README [no ci] ( #7335 )
2024-05-17 10:11:03 +10:00
24ecb58168
Revert "server bench: fix bench not waiting for model load ( #7284 )" ( #7334 )
...
This reverts commit 583fd6b000
2024-05-16 20:43:45 +02:00
9afdffe70e
rpc : get available mem for the CPU backend
...
This can be overridden with the -m command line option
ref: #7293
2024-05-16 12:04:08 +03:00
3b3963c55c
rpc : add command line arg for specifying backend memory
...
ref: #7293
2024-05-16 09:58:29 +03:00
ad52d5c259
doc: add references to hugging face GGUF-my-repo quantisation web tool. ( #7288 )
...
* chore: add references to the quantisation space.
* fix grammer lol.
* Update README.md
Co-authored-by: Julien Chaumond <julien@huggingface.co>
* Update README.md
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Julien Chaumond <julien@huggingface.co>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2024-05-16 15:38:43 +10:00
344f9126cc
ggml : tag ggml_tensor::backend as deprecated ( #7290 )
2024-05-15 15:08:48 +02:00
ea3b0590ee
embedding : free the batch after execution ( #7297 )
2024-05-15 15:01:12 +03:00
583fd6b000
server bench: fix bench not waiting for model load ( #7284 )
2024-05-15 08:44:16 +02:00
4f0263633b
server: free sampling contexts on exit ( #7264 )
...
* server: free sampling contexts on exit
This cleans up last leak found by the address sanitizer.
* fix whitespace
* fix whitespace
2024-05-14 16:11:24 +02:00
1265c670fd
Revert "move ndk code to a new library ( #6951 )" ( #7282 )
...
This reverts commit efc8f767c8
2024-05-14 16:10:39 +03:00
5e31828d3e
ggml : add RPC backend ( #6829 )
...
* ggml : add RPC backend
The RPC backend proxies all operations to a remote server which runs a
regular backend (CPU, CUDA, Metal, etc).
* set TCP_NODELAY
* add CI workflows
* Address review comments
* fix warning
* implement llama_max_devices() for RPC
* Address review comments
* Address review comments
* wrap sockfd into a struct
* implement get_alignment and get_max_size
* add get_device_memory
* fix warning
* win32 support
* add README
* readme : trim trailing whitespace
* Address review comments
* win32 fix
* Address review comments
* fix compile warnings on macos
2024-05-14 14:27:19 +03:00
efc8f767c8
move ndk code to a new library ( #6951 )
2024-05-14 17:30:30 +10:00
27f65d6267
docs: Fix typo and update description for --embeddings flag ( #7026 )
...
- Change '--embedding' to '--embeddings' in the README
- Update the description to match the latest --help output
- Added a caution about defining physical batch size
2024-05-14 15:20:47 +10:00
30e70334f7
llava-cli: fix base64 prompt ( #7248 )
2024-05-14 00:02:36 +10:00
1c570d8bee
perplexity: add BF16 vs. FP16 results ( #7150 )
2024-05-13 13:03:27 +02:00
e586ee4259
change default temperature of OAI compat API from 0 to 1 ( #7226 )
...
* change default temperature of OAI compat API from 0 to 1
* make tests explicitly send temperature to OAI API
2024-05-13 12:40:08 +10:00
Oleksandr Kuvshynov
Amir
jaime-m-p
Johannes Gäßler
Georgi Gerganov
Fred Douglas
strawberrymelonpanda
Radoslav Gerganov
Leon Knauer
Pierrick Hymbert
Vaibhav Srivastav
slaren
dm4
Steve Grubb
Brian
Elton Kola
Ryuei
k.h.lai
Benjamin Findley