e52a7aa304
Given that default HTTPServer handles only one connection and inturn request at any given time, so if a client opens connection and then doesnt do anything with it, it will block other clients by putting their requests into network queue for long. So to overcome the above issue switch to ThreadingHTTPServer, which starts a new thread for each request. Given that previously ssl wrapping was done wrt the main server socket, even with switching to ThreadingHTTPServer, the handshake for ssl/tls still occurs in the main thread before a child thread is started for parallel request handling, thus the ssl handshake phase blocking other client requests. So now avoid wrapping ssl wrt the main server socket, instead wait for ThreadingHttpServer to start the new thread for a client request ie after a connection is accepted for the client, before trying to wrap the connection in ssl. This ensures that the ssl handshake occurs in this child (ie client request related) thread. So some rogue entity opening a http connection and not doing ssl handshake wont block. Inturn in this case the rfile and wfile instances within the proxy handler need to be remapped to the new ssl wrapped socket. |
||
|---|---|---|
| .. | ||
| batched-bench | ||
| cvector-generator | ||
| export-lora | ||
| gguf-split | ||
| imatrix | ||
| llama-bench | ||
| main | ||
| mtmd | ||
| perplexity | ||
| quantize | ||
| rpc | ||
| run | ||
| server | ||
| tokenize | ||
| tts | ||
| CMakeLists.txt | ||