4 Commits

Author	SHA1	Message	Date
Xuan-Son Nguyen	ddcb75dd8a	server: add auto-sleep after N seconds of idle (#18228) ... * implement sleeping at queue level * implement server-context suspend * add test * add docs * optimization: add fast path * make sure to free llama_init * nits * fix use-after-free * allow /models to be accessed during sleeping, fix use-after-free * don't allow accessing /models during sleep, it is not thread-safe * fix data race on accessing props and model_meta * small clean up * trailing whitespace * rm outdated comments	2025-12-21 02:24:42 +01:00
Xuan-Son Nguyen	951520ddb0	server: delegate result_state creation to server_task (#17835) ... * server: delegate result_state creation to server_task * remove unued states * add more docs	2025-12-08 17:04:38 +01:00
Xuan-Son Nguyen	f896d2c34f	server: improve speed of speculative decoding (#17808) ... * server: improve speed of speculative decoding * fix small draft case * add link to the PR * server : fix generation time measurement * server : fix draft acceptance logs (add SRV_CNT, SLT_CNT macros) * server : add comment * add PR to docs --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2025-12-08 14:35:28 +01:00
Xuan-Son Nguyen	37a4f63244	server : add development documentation (#17760) ... * first draft * rewrite * update & remove duplicated sections	2025-12-08 13:54:58 +01:00