happyz
/
llama.cpp
mirror of https://github.com/ggerganov/llama.cpp.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

SimpleSallap:SimpleProxy: MultiThreading 

Given that default HTTPServer handles only one connection and inturn
request at any given time, so if a client opens connection and then
doesnt do anything with it, it will block other clients by putting their
requests into network queue for long.

So to overcome the above issue switch to ThreadingHTTPServer, which
starts a new thread for each request.

Given that previously ssl wrapping was done wrt the main server socket,
even with switching to ThreadingHTTPServer, the handshake for ssl/tls
still occurs in the main thread before a child thread is started for
parallel request handling, thus the ssl handshake phase blocking other
client requests.

So now avoid wrapping ssl wrt the main server socket, instead wait for
ThreadingHttpServer to start the new thread for a client request ie
after a connection is accepted for the client, before trying to wrap
the connection in ssl. This ensures that the ssl handshake occurs in
this child (ie client request related) thread. So some rogue entity
opening a http connection and not doing ssl handshake wont block.

Inturn in this case the rfile and wfile instances within the proxy
handler need to be remapped to the new ssl wrapped socket.
This commit is contained in:
hanishkvc 2025-12-05 01:53:12 +05:30
parent c4e0c03107
commit e52a7aa304
1 changed files with 12 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
tools/server/public_simplechat/local.tools/simpleproxy.py
View File

@ -36,7 +36,8 @@ gMe = {
'--config': '/dev/null',
'--debug': False,
'bearer.transformed.year': "",
'server': None
'server': None,
'sslContext': None,
}
gConfigType = {
@ -164,6 +165,14 @@ class ProxyHandler(http.server.BaseHTTPRequestHandler):
def handle(self) -> None:
print(f"\n\n\nDBUG:ProxyHandler:Handle:RequestFrom:{self.client_address}")
try:
if (gMe['sslContext']):
self.request = gMe['sslContext'].wrap_socket(self.request, server_side=True)
self.rfile = self.request.makefile('rb', self.rbufsize)
self.wfile = self.request.makefile('wb', self.wbufsize)
except:
print(f"ERRR:ProxyHandler:SSLHS:{traceback.format_exception_only(sys.exception())}")
return
return super().handle()
@ -279,13 +288,13 @@ def setup_server():
"""
try:
gMe['serverAddr'] = ('', gMe['--port'])
gMe['server'] = http.server.HTTPServer(gMe['serverAddr'], ProxyHandler)
gMe['server'] = http.server.ThreadingHTTPServer(gMe['serverAddr'], ProxyHandler)
if gMe.get('--sec.keyfile') and gMe.get('--sec.certfile'):
sslCtxt = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
sslCtxt.load_cert_chain(certfile=gMe['--sec.certfile'], keyfile=gMe['--sec.keyfile'])
sslCtxt.minimum_version = ssl.TLSVersion.MAXIMUM_SUPPORTED
sslCtxt.maximum_version = ssl.TLSVersion.MAXIMUM_SUPPORTED
gMe['server'].socket = sslCtxt.wrap_socket(gMe['server'].socket, server_side=True)
gMe['sslContext'] = sslCtxt
print(f"INFO:SetupServer:Starting on {gMe['serverAddr']}:Https mode")
else:
print(f"INFO:SetupServer:Starting on {gMe['serverAddr']}:Http mode")