move the actual chat handshake with ai server into a seperate code
to an extent.
also initial anchor to trap handshake http error responses
Rather come to think of it, its better to move this into SimpleChat
class.
Use finally to ensure any needed cleanup for handle_user_submit
occurs within itself.
Update the descriptions of set and get to indicate the possible
corner cases or rather semantic in such situations.
Update the readme also a bit. The auto save and restore mentioned
has nothing to do with the new data store mechanism.
In the eagerness of initial skeleton, had forgotten that the
root/generic tool call router takes care of parsing the json string
into a object, before calling the tool call, so no need to try
parse again. Fixed the same.
Hadnt converted the object based response from data store related
calls in the db web worker, into json string before passing to the
generic tool response callback, fixed the same.
- Rather the though of making the ChatMsgEx.createAllInOne handle
string or object set aside for now, to keep things simple and
consistant to the greatest extent possible across different flows.
And good news - flow is working atleast for the overall happy path
Need to check what corner cases are lurking like calling set on
same key more than once, seemed to have some flow oddity, which I
need to check later.
Also maybe change the field name to value from data in the response
to get, to match the field name convention of set. GPT-OSS is fine
with it. But worst case micro / nano / pico models may trip up, in
worst case, so better to keep things consistent.
So mention that may be ai can send complex objects in stringified
form. Rather once type of value is set to string, ai should normally
do it, but no harm is hinting.
Create the DB store
Try Get and Set operations
The post back to main thread done from asynchronous paths.
NOTE: given that it has been ages since indexed db was used,
so this is a logical implementation by refering to mdn as needed.
Define rules to ensure that chat message contents wrap so as to
avoid overflowing beyond the size of the screen being viewed.
The style used for chat message role to be placed with vertical
oriented text adjacent to the actual message content on the side
seems to be creating issue with blank pages in some browsers,
so avoid that styling when one is printing.
Seperate out the message ui block into a container containing
a role block and contents container block.
This will allow themeing of these seperately, if required.
As part of same, currently the role has been put to the side
of the message with vertical text flow.
Add a newline between name and content in the xml representation
of the tool response, so that it is more easy to distinguish things
Add github, linkedin and apnews domains to allowed.domains for
simpleproxy.py
Fix up the initial skeleton / logic as needed.
Remember that we are working with potentially a subset of chat
messages from the session, given the sliding window logic of
context managing on client ui side, so fix up the logic to use
the right subset of messages array and not the global xchat
when deciding whether a message is the last or last but one,
which need special handling wrt Assistant (with toolcall) and
Tool (ie response) messages.
Moving tool call ui setup as well as tool call response got ui
setup into ChatShow of MultiChatUI ensures that switching between
chat sessions handle the ui wrt tool call triggering ui and tool
call response submission related ui as needed properly.
Rather even loading a previously auto saved chat session if it had
tool call or tool call response to be handled, the chat ui will be
setup as needed to continue that session properly.
Rather simplify and make the content_equiv provide a relatively
simple and neat representation of the reasoning with content and
toolcall as the cases may be.
Also remove the partial new para that I had introduced in the
initial go for reasoning.
Pass chatId to tool call, and use chatId in got tool call resp,
to decide as to to which chat session the async tool call resp
belongs and inturn if auto submit timer should be started if auto
is enabled.
Update the immidiate tool call triggering failure and tool call
response timeout paths to use the new ToolTemp and MultiChatUI
based chat show logics.
Actual tool call itself generating errors, is already handled
in the previous commit changes.
Add a new role ToolTemp, which is used to maintain any tool call
response on the client ui side, without submitting it to the server
ie till user or auto submit triggers the submitting of that tool
call response.
When ever a tool call response is got, create a ToolTemp role based
message in the corresponding chat session. And dont directly update
the user query input area, rather leave it to the updated simplechat
show and the new multichatui chat_show helper and inturn whether the
current chat session active in ui is same as the one for which the
tool call response has been recieved.
TODO: Currently the response message is added to the current
active chat session, but this needs to be changed by tracking
chatId/session through the full tool call cycle and then adding
the tool call response in the related chat session, and inturn
updating or not the ui based on whether that chat session is
still the active chat session in ui or not, given that tool call
gets handled in a asynchronous way.
Now when that tool call response is submitted, promote the equiv
tool temp role based message that should be in the session's chat
history as the last message into becoming a normal tool response
message.
SimpleChat.show has been updated to take care of showing any
ToolTemp role message in the user query input area.
A newer chat_show helper added to MultiChatUI, that takes care of
calling SimpleChat.show, provided the chat_show is being requested
for the currently active in ui, chat session. As well as to take
care of passing both the ChatDiv and elInUser. Converts users of
SimpleChat.show to use MultiChatUI.chat_show
instead of using the shared bearer token as is, hash it with
current year and use the hash.
keep /aum path out of auth check.
in future bearer token could be transformed more often, as well as
with additional nounce/dynamic token from server got during initial
/aum handshake as also running counter and so ...
NOTE: All these circus not good enough, given that currently the
simpleproxy.py handshakes work over http. However these skeletons
put in place, for future, if needed.
TODO: There is a once in a bluemoon race when the year transitions
between client generating the request and server handling the req.
But other wise year transitions dont matter bcas client always
creates fresh token, and server checks for year change to genrate
fresh token if required.
Add a config entry called bearer.insecure which will contain a
token used for bearer auth of http requests
Make bearer.insecure and allowed.domains as needed configs, and
exit program if they arent got through cmdline or config file.
Ensure load_config gets called on encountering --config in cmdline,
so that the user has control over whether cmdline or config file
will decide the final value of any given parameter.
Ensure that str type values in cmdline are picked up directly, without
running them through ast.literal_eval, bcas otherwise one will have to
ensure throught the cmdline arg mechanism that string quote is retained
for literal_eval
Have the """ function note/description below def line immidiately
so that it is interpreted as a function description.
Now both follow a similar mechanism and do the following
* exit on finding any issue, so that things are in a known
state from usage perspective, without any confusion/overlook
* check if the cmdlineArgCmd/configCmd being processed is a known
one or not.
* check value of the cmd is of the expected type
* have a generic flow which can accomodate more cmds in future
in a simple way
Moved it into Me->tools, so that end user can modify the same as
required from the settings ui.
TODO: Currently, if tc response is got after a tool call timed out
and user submitted default timed out error response, the delayed
actual response when it is got may overwrite any new content in
user query box, this needs to be tackled.
Avoid code duplication, by creating helpers for setup and toolcall.
Also send indication of the path that will be used, when checking
for simpleproxy.py server to be running at runtime setup.
Initial go at implementing a web search tool call, which uses the
existing UrlText support of the bundled simpleproxy.py.
It allows user to control the search engine to use, by allowing
them to set the search engine url template.
The logic comes with search engine url template strings for
duckduckgo, brave, bing and google. With duckduckgo set by default.
include info about the auto option within tools.
use nonwrapped text wrt certain sections, so that the markdown
readme can be viewed properly wrt the structure of the content
in it.
Instead of enforcing always explicit user triggered tool calling,
now user is given the option whether to use explicit user triggered
tool calling or to use auto triggering after showing tool details
for a user specified amount of seconds.
NOTE: The current logic doesnt account for user clicking the buttons
before the autoclick triggers; need to cancel the auto clicks, if
user triggers before autoclick, ie in future.
bing raised a challenge for chrome triggered search requests after
few requests, which were spread few minutes apart, while still
seemingly allowing wget based search to continue (again spread
few minutes apart).
Added a simple helper to trace this, use --debug True to enable
same.
Instead of simple concatenating of tool call id, name and result
now use browser's dom logic to create the xml structure used for
now to store these within content field.
This should take care of transforming / escaping any xml special
chars in the result, so that extracting them later for putting
into different fields in the server handshake doesnt have any
problem.
Use DOMParser parseFromString in text/html mode rather than text/xml
as it makes it more relaxed without worrying about special chars
of xml like & etal
mimicing got req in generated req helps with duckduckgo also and
not just yahoo.
also update allowed.domains to allow a url generated by ai when
trying to access the bing's news aggregation url
The tagging of messages wrt ValidateUrl and UrlReq
Also dump req
Move check for --allowed.domains to ValidateUrl
NOTE: Also with mimicing of user agent etal from got request to
the generated request, yahoo search/news is returning results now,
instead of the bland error before.
with allowed domains set to few sites in general to show its use
this includes some sites which allow search to be carried out
through them as well as provide news aggregation
Had confused between js and python wrt accessing dictionary
contents and its consequence on non existent key. Fixed it.
Use different error ids to distinguish between failure in common
urlreq and the specific urltext and urlraw helpers.
Also remove more inner/detailed stuff from show info in not bAll
mode, given that many of the previous differentiated stuff have
been moved into chatProps and inturn shown for now
Fix up the oversights wrt any depth trapping flow
Remember to start the propWithTree being checked/trapped with :
to indicate the root of the prop hierarchy and also use : as sep
between the elements of the props hierarchy tree
Also had forgotten about the goof up possible with using in in a
condition statement to check for array to contain a entry of interest
in JS, fixed it now.
Maintain the current property hierarchy to its root over recursive
calls.
Allow callers to specify the props to be trapped using the prop
hierarchy.
Pass the prop hierarchy to the fTrapper.
This should allow one to trap any prop wrt its editing ui setup,
irrespective of whether it is a prop of the main object passed,
or a member of a child prop of the main object passed or so ...
Update the setting up of ChatHistoryInCtxt and ApiEndPoint to follow
the new semantic/flow.
NOTE: not a robust check, just tries to establish a http connection
for now and doesnt really check if it is the specific proxy srvr
of interest or not.
Take care of the possibility of content not being there as well as
take care of retrieving the tool calls for further processing.
With this tool calls should work in non streaming mode also
Make the previously relatively generic flow wrt apiRequestOptions
settings into a fully generic reusable by others flow.
Rather had stopped short of it, when previously moved onto other
things at that time.
So that the same error path is used for logical error wrt http req
also, without needing a different path for it.
Dont forget to return the resp text/json/..., so that the contents
are passed along the promise then chain
Add a new send headers common helper and use the same wrt the
overridden send_error as well as do_OPTIONS
This ensures that if there is any error during proxy opertions,
the send_error propogates to the fetch from any browser properly
without browser intercepting it with a CORS error
Ensures that if the url being requested as any query strings in
them then things dont get messed up, when the url to get inc its
query is extracted from the proxy request's query string
First identify lines which have only whitespace and replace them
with lines with only newline char in them.
Next strip out adjacent lines, if they have only newlines
As html can be malformed, xml ElementTree XMLParser cant handle
the same properly, so switch to the HtmlParser helper class that is
provided by python and try extend it.
Currently a minimal skeleton to just start it out, which captures
only the body contents.
Declare the result of UrlReq as a DataClass, so that one doesnt
goof up wrt updating and accessing members.
Duplicate UrlRaw into UrlText, need to add Text extracting from
html next for UrlText
Dont forget to map members of got entity from fetch to things
from saved original promise, bcas remember what is got is a promise.
also
add some comments around certain decisions and needed exploration
also possible refinement wrt trapping, if needed, added as comment
all or allSettled to use or not is the question.
whether to wait for a round trip through the related event loop or
not is also a question.
Also update the sliding window context size to last 9 chat messages
so that there is a sufficiently large context for multi turn tool
calls based adjusting by ai and user, without needing to go full
hog, which has the issue of overflowing the currently set context
window wrt the loaded ai model.
these common helpers avoid needing ignore tagging to ts-check, in
places where valid constructs have been used which go beyond strict
structured js handling that is tried to be achieved using it, but
are still valid and legal.
Expand the xml format id, name and content in content field of
tool result into apropriate fields in the tool result message sent
to the genai/llm engine on the server.
Use HTMLElement's dataset to maintain tool call id along with
the element which maintains the toolname.
Pass it along to the tools manager and inturn the actual tool
calls and through them to the web worker handling the tool call
related code and inturn returning it back as part of the obj
which is used to return the tool call result.
Embed the tool call id, function name and function result into
the content field of chat message in terms of a xml structure
Also make use of tool role to send back the tool call result.
Do note that currently the id, name and content are all embedded
into the content field of the tool role message sent to the
ai engine on the server.
NOTE: Use the user query entry area for showing tool call result
in the above mentioned xml form, as well as for user to enter
their own queries. Based on presence of the xml format data at
beginning the logic will treat it has a tool result and if not
then as a normal user query.
The css has been updated to help show tool results/msgs in a
lightyellow background
Users of recent_chat updated to work with ChatMessageEx
As part of same recent_chat_ns also added, for the case where the
array of chat messages can be passed as is ie in the chat mode,
provided it has only the network handshake representation of the
messages.
Simplify Add semantic by expecting any validation of stuff before
adding to be done by the callers of Add and not by add itself.
Also update it to expect ChatMessageEx object
Update all users of add to follow the new syntax and semantic.
Remove the old and ununsed AddSysPromptOnlyAtBegin helper
GetSystemLatest and its users updated wrt ChatMessageEx.
RecentChat updated wrt ChatMessageEx. Also now irrespective of
whether full history is being retrieved or only a subset, both
cases refer to the ChatMessageEx instances in SimpleChat.xchat
without creating new instances of anything.
Use the equivalent update_stream directly added to ChatMessageEx.
update_stream is also more generic to some extent and also directly
implemented by the ChatMessageEx class.
Rename ChatMessage to ChatMessageEx.
Add typedefs for NSToolCall and NSChatMessage, they represent the
way the corresponding data is structured in network hs.
Add logic to build the ChatMessageEx from data got over network in
streaming mode.
As the tool calling, if enabled, will need access to last few
user query and ai assistant responses (which will also include
in them the tool call requests and the corresponding results),
so that the model can build answers based on its tool call reqs
and got responses, and also given that most of the models these
days have sufficiently large context windows, so the sliding
window context implemented by SimpleChat logic has been increased
by default to include last 4 query and their responses roughlty.
Had forgotten to specify type as module wrt web worker, in order
to allow it to import the toolsconsole module.
Had forgotten to maintain the id of the timeout handler, which is
needed to clear/stop the timeout handler from triggering, if tool
call response is got well in time.
As I am currently reverting the console redirection at end of
handling a tool call code in the web worker message handler, I
need to setup the redirection each time. Also I had forgotten
to clear the console.log capture data space, before a new tool
call code is executed, this is also fixed by this change.
TODO: Need to abort the tool call code execution in the web worker
if possible in future, if the client / browser side times out
waiting for tool call response, ie if the tool call code is taking
up too much time.
tools manager/module
* setup the web worker that will help execute the tool call related
codes in a js environment that is isolated from the browsers main
js environment
* pass the web worker to the tool call providers, for them to use
* dont wait for the result from the tool call, as it will be got
later asynchronously through a message
* allow users of the tools manager to register a call back, which
will be called when ever a message is got from the web worker
containing response wrt previously requested tool call execution.
simplechat
* decouple toolcall response handling and toolcall requesting logic
* setup a timeout to take back control if tool call takes up too
much time. Inturn help alert the ai model, that the tool call
took up too much time and so was aborted, by placing a approriate
tagged tool response into user query area.
* register a call back that will be called when response is got
asynchronously wrt anye requested tool calls.
In turn take care of updating the user query area with response
got wrt the tool call, along with tool response tag around it.
These no longer need to worry about
* setting up the console.log related redirection to capture
the generated outputs, nor about
* setting up a dynamic function for executing the needed
tool call related code
The web worker setup to help run tool calls in a relatively
isolated environment independent of the main browser env,
takes care of these.
One needs to only worry about getting the handle to the
web worker to use and inturn pass the need code wrt the
tool call to it.
The request for code to run as well as the resultant response data
both need to follow a structured object convention, so that it is
easy to map a request and the corresponding response to some extent.
Try ensure as well as verify that original console.log is saved
and not overwritten. Throw an exception if things seem off wrt
same.
Also ensure to add a newline at end of console.log messages
Instead of automatically calling any requested tool by the GenAi
/ llm, that is from the tail end of the handle user submit btn
click,
Now if the GenAi/LLM has requested any tool to be called, then
enable the Tool Run related UI elements and fill them with the
tool name and tool args.
In turn the user can verify if they are ok with the tool being
called and the arguments being passed to it. Rather they can
even fix any errors in the tool usage like the arithmatic expr
to calculate that is being passed to simple_calculator or the
javascript code being passed to run_javascript_function_code
If user is ok with the tool call being requested, then trigger
the same.
The results if any will be automatically placed into the user
query text area.
User can cross verify if they are ok with the result and or
modify it suitabley if required and inturn submit the same to
the GenAi/LLM.
Instead of automatically calling the requested tool with supplied
arguments, rather allow user to verify things before triggering the
tool.
NOTE: User already provided control over tool_response before
submitting it to the ai assistant.
So that when tool handler writes the result to the tc_switch, it
can make use of the same, to write to the right location.
NOTE: This also fixes the issue with I forgetting to rename the
key in js_run wrt writing of result.
Move tool calling logic into tools module.
Try trap async promise failures by awaiting results of tool calling
and putting full thing in an outer try catch. Have forgotten the
nitty gritties of JS flow, this might help, need to check.
As output generated by any tool/function call is currently placed
into the TextArea provided for End user (for their queries), bcas
the GenAi (engine/LLM) may be expecting the tool response to be
sent as a user role data with tool_response tag surrounding the
results from the tool call. So also now at the end of submit btn
click handling, the end user input text area is not cleared, if
there was a tool call handled, for above reasons.
Also given that running a simple arithmatic expression in itself
doesnt generate any output, so wrap them in a console.log, to
help capture the result using the console.log trapping flow that
is already setup.
Checks for toolname to be defined or not in the GenAi's response
If toolname is set, then check if a corresponding tool/func exists,
and if so call the same by passing it the GenAi provided toolargs
as a object.
Inturn the text generated by the tool/func is captured and put
into the user input entry text box, with tool_response tag around
it.
The implementations of javascript and simple_calculator now use
provided helpers to trap console.log messages when they execute
the code / expression provided by GenAi and inturn store the
captured log messages in the newly added result key in tc_switch
This should help trap the output generated by the provided code
or expression as the case maybe and inturn return the same to the
GenAi, for its further processing.
Previously if content was empty, it would have always sent the
toolcall info related version even if there was no toolcall info
in it. Fixed now to return empty string, if both content and
toolname are empty.
As there could be failure wrt getting the response from the ai
server some where in between a long response spread over multiple
parts, the logic uses the latestResponse to cache the response
as it is being received. However once the full response is got,
one needs to transfer it to a new instance of AssistantResponse
class, so that latestResponse can be cleared, while the new
instance can be used in other locations in the flow as needed.
Achieve the same now.
Switch oneshot handler to use AssistantResponse, inturn currenlty
only handle the normal content in the response.
TODO: If any tool_calls in the oneshot response, it is currently
not handled.
Inturn switch the generic/toplevel handle response logic to use
AssistantResponse class, given that both oneshot and the
multipart/streaming flows use/return it.
Inturn add trimmedContent member to AssistantResponse class and
make the generic handle response logic to save the trimmed content
into this. Update users of trimmed to work with this structure.
Make latestResponse into a new class based type instance wrt
ai assistant response, which is what it represents.
Move clearing, appending fields' values and getting assistant's
response info (irrespective of a content or toolcall response)
into this new class and inturn use the same.
I was wrongly checking for finish_reason to be non null, before
trying to extract the genai content/toolcalls, have fixed this
oversight with the new flow in progress.
I had added few debug logs to identify the above issue, need to
remove them later. Note: given that debug logs are disabled by
replacing the debug function during this program's initialisation,
which I had forgotten about, I didnt get the debug messages and
had to scratch my head a bit, before realising this and the other
issue ;)
Also either when I had originally implemented simplechat 1+ years
back, or later due to changes on the server end, the streaming
flow sends a initial null wrt the content, where it only sets the
role. This was not handled in my flow on the client side, so a
null was getting prepended to the chat messages/responses from the
server. This has been fixed now in the new generic flow.
Update response_extract_stream to check for which field is being
currently streamed ie is it normal content or tool call func name
or tool call func args and then return the field name and extracted
value.
Previously it was always assumed that only normal content will be
returned.
Currently it is assumed that the server will only stream one of the
3 supported fields at any time and not more than one of them at the
same time.
TODO: Have to also add logic to extract the reasoning field later,
ie wrt gen ai models which give out their thinking.
Have updated append_response to expect both the key and the value
wrt the latestResponse object, which it will be manipualted.
Previously it was always assumed that content is what will be got
and inturn appended.
Changed latestResponse type to an object instead of a string.
Inturn it contains entries for content, toolname and toolargs.
Added a custom clear logic due to the same and used it to replace
the previously simple assigning of empty string to latestResponse.
For now in all places where latestReponse is used, I have replaced
with latestReponse.content.
Next need to handle identifying the field being streamed and inturn
append to it. Also need to add logic to call tool, when tool_call
triggered by genai.
Define the meta that needs to be passed to the GenAi Engine.
Define the logic that implements the tool call, if called.
Implement the flow/structure such that a single tool calls
implementation file can define multiple tool calls.
Enable streaming by default, to check the handshake before going
on to change the code, given that havent looked into this for more
than a year now and have been busy with totally different stuff.
Also updated the user messages used for testing a bit
Previously, cmake was forcing `_WIN32_WINNT=0x0A00` for MinGW builds,
This caused "macro redefined" warnings with toolchains that define the version.
This also removes the `GGML_WIN_VER` variable as it is no longer needed.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* webui: Fix zero pasteLongTextToFileLen to disable conversion being overridden
Zero pasteLongTextToFileLen should disable the conversion, but it was
overwritten with 2500.
* Apply suggestions from code review
* Update webui build
* llama-server: add router multi-model tests (#17704)
Add 4 test cases for model router:
- test_router_unload_model: explicit model unloading
- test_router_models_max_evicts_lru: LRU eviction with --models-max
- test_router_no_models_autoload: --no-models-autoload flag behavior
- test_router_api_key_required: API key authentication
Tests use async model loading with polling and graceful skip when
insufficient models available for eviction testing.
utils.py changes:
- Add models_max, models_dir, no_models_autoload attributes to ServerProcess
- Handle JSONDecodeError for non-JSON error responses (fallback to text)
* llama-server: update test models to new HF repos
* add offline
* llama-server: fix router LRU eviction test and add preloading
Fix eviction test: load 2 models first, verify state, then load
3rd to trigger eviction. Previous logic loaded all 3 at once,
causing first model to be evicted before verification could occur.
Add module fixture to preload models via ServerPreset.load_all()
and mark test presets as offline to use cached models
* llama-server: fix split model download on Windows
---------
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
* common : implement parser combinators to simplify chat parsing
* add virtual destructor to parser_base
* fix memory leak from circular references of rules
* implement gbnf grammar building
* remove unused private variable
* create a base visitor and implement id assignment as a visitor
* fix const ref for grammar builder
* clean up types, friend classes, and class declarations
* remove builder usage from until_parser
* Use a counter class to help assign rule ids
* cache everything
* add short description for each parser
* create a type for the root parser
* implement repetition parser
* Make optional, one_or_more, and zero_or_more subclasses of repetition
* improve context constructor
* improve until parsing and add benchmarks
* remove cached() pattern, cache in parser_base with specialized parsing functions for each parser
* improve json parsing performance to better match legacy parsing
* fix const auto * it for windows
* move id assignment to classes instead of using a visitor
* create named rules in the command r7b example
* use '.' for any in GBNF
* fix parens around choices in gbnf grammar
* add convenience operators to turn strings to literals
* add free-form operators for const char * to simplify defining literals
* simplify test case parser
* implement semantic actions
* remove groups in favor of actions and a scratchpad
* add built in actions for common operations
* add actions to command r7b example
* use std::default_searcher for platforms that don't have bm
* improve parser_type handling and add cast helper
* add partial result type to better control when to run actions
* fix bug in until()
* run actions on partial results by default
* use common_chat_msg for result
* add qwen3 example wip
* trash partial idea and simplify
* move action arguments to a struct
* implement aho-corasick matcher for until_parser and to build exclusion grammars
* use std::string for input, since std::string_view is incompatible with std::regex
* Refactor tests
* improve qwen3 example
* implement sax-style parsing and refactor
* fix json string in test
* rename classes to use common_chat_ prefix
* remove is_ suffix from functions
* rename from id_counter to just counter
* Final refactored tests
* Fix executable name and editorconfig-checker
* Third time's the charm...
* add trigger parser to begin lazy grammar rule generation
* working lazy grammar
* refactor json rules now that we check for reachability
* reduce pointer usage
* print out grammars in example
* rename to chat-peg-parser* and common_chat_peg_parser*
* Revert unrelated changes
* New macros for CMakeLists to enable multi-file compilations
* starting unicode support
* add unicode support to char_parser
* use unparsed args as additional sources
* Refactor tests to new harness
* Fix CMakeLists
* fix rate calculation
* add unicode tests
* fix trailing whitespace and line endings
skip-checks: true
* Helpers + rewrite qwen3 with helpers
* Fix whitespace
* extract unicode functions to separate file
* refactor parse unicode function
* fix compiler error
* improve construction of sequence/choice parsers
* be less clever
* add make_parser helper function
* expand usage of make_parser, alias common_chat_msg_peg_parser_builder to builder in source
* lower bench iterations
* add unicode support to until_parser
* add unicode support to json_string_parser
* clean up unicode tests
* reduce unicode details to match src/unicode.cpp
* simplify even further
* remove unused functions
* fix type
* reformat char class parsing
* clean up json string parser
* clean up + fix diagnostics
* reorder includes
* compact builder functions
* replace action_parser with capture_parser, rename env to semantics
* rename env to semantics
* clean up common_chat_parse_context
* move type() to below constant
* use default constructor for common_chat_peg_parser
* make all operators functions for consistency
* fix compilation errors in test-optional.cpp
* simplify result values
* rename json_string_unquoted to json_string_content
* Move helper to separate class, add separate explicit and helper classes
* Whitespace
* Change + to append()
* Reformat
* Add extra helpers, tests and Minimax example
* Add some extra optional debugging prints + real example of how to use them
* fix bug in repetitions when min_count = 0 reports failures
* dump rule in debug
* fix token accumulation and assert parsing never fails
* indent debug by depth
* use LOG_* in tests so logs sync up with test logs
* - Add selective testing
- Refactor all messaging to use LOG_ERR
- Fix lack of argument / tool name capturing
- Temporary fix for double event capture
* refactor rule() and introduce ref()
* clean up visitor
* clean up indirection in root parser w.r.t rules
* store shared ptr directly in parser classes
* replace aho-corasick automation with a simple trie
* Reset prev for qwen3 helper example variant
* refactor to use value semantics with std::variant/std::visit
* simplify trie_matcher result
* fix linting issues
* add annotations to rules
* revert test workaround
* implement serializing the parser
* remove redundant parsers
* remove tests
* gbnf generation fixes
* remove LOG_* use in tests
* update gbnf tests to test entire grammar
* clean up gbnf generation and fix a few bugs
* fix typo in test output
* remove implicit conversion rules
* improve test output
* rename trie_matcher to trie
* simplify trie to just know if a node is the end of a word
* remove common_chat_ prefix and ensure a common_peg_ prefix to all types
* rename chat-peg-parser -> peg-parser
* promote chat-peg-parser-helper to chat-peg-parser
* checkpoint
* use a static_assert to ensure we handle every branch
* inline trivial peg parser builders
* use json strings for now
* implement basic and native chat peg parser builders/extractors
* resolve refs to their rules
* remove packrat caching (for now)
* update tests
* compare parsers with incremental input
* benchmark both complete and incremental parsing
* add raw string generation from json schema
* add support for string schemas in gbnf generation
* fix qwen example to include \n
* tidy up example
* rename extractor to mapper
* rename ast_arena to ast
* place basic tests into one
* use gbnf_format_literal from json-schema-to-grammar
* integrate parser with common/chat and server
* clean up schema and serialization
* add json-schema raw string tests
* clean up json creation and remove capture parser
* trim spaces from reasoning and content
* clean up redundant rules and comments
* rename input_is_complete to is_partial to match rest of project
* simplify json rules
* remove extraneous file
* remove comment
* implement += and |= operators
* add comments to qwen3 implementation
* reorder arguments to common_chat_peg_parse
* remove commented outdated tests
* add explicit copy constructor
* fix operators and constness
* wip: update test-chat for qwen3-coder
* bring json parser closer to json-schema-to-grammar rules
* trim trailing space for most things
* fix qwen3 coder rules w.r.t. trailing spaces
* group rules
* do not trim trailing space from string args
* tweak spacing of qwen3 grammar
* update qwen3-coder tests
* qwen3-coder small fixes
* place parser in common_chat_syntax to simplify invocation
* use std::set to collect rules to keep order predictable for tests
* initialize parser to make certain platforms happy
* revert back to std::unordered_set, sort rule names at the end instead
* uncomment rest of chat tests
* define explicit default constructor
* improve arena init and server integration
* fix chat test
* add json_member()
* add a comprehensive native example
* clean up example qwen test and add response_format example to native test
* make build_peg_parser accept std::function instead of template
* change peg parser parameters into const ref
* push tool call on tool open for constructed parser
* add parsing documentation
* clean up some comments
* add json schema support to qwen3-coder
* add id initializer in tests
* remove grammar debug line from qwen3-coder
* refactor qwen3-coder to use sequence over operators
* only call common_chat_peg_parse if appropriate format
* simplify qwen3-coder space handling
* revert qwen3-coder implementation
* revert json-schema-to-grammar changes
* remove unnecessary forward declaration
* small adjustment to until_parser
* rename C/C++ files to use dashes
* codeowners : add aldehir to peg-parser and related files
---------
Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
* Revert "rm unused fn"
This reverts commit f2dbe9c087.
* server: explicitly set exec path when create new instance
* put back TODO
* only call get_server_exec_path() once
* add fallback logic
* Adjust to pytorch
* Add antialiasing upscale
* Increase number of patches to 1024
* Handle default marker insertion for LFM2
* Switch to flag
* Reformat
* Cuda implementation of antialias kernel
* Change placement in ops.cpp
* consistent float literals
* Pad only for LFM2
* Address PR feedback
* Rollback default marker placement changes
* Fallback to CPU implementation for antialias implementation of upscale
* git mv
* add server-context.h
* add server-context.h
* clean up headers
* cont : cleanup
* also expose server_response_reader (to be used by CLI)
* fix windows build
* decouple server_routes and server_http
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
As [1] explained, the real debug message will be like:
"res operator(): operator() : queue result stop"
Set the name explicitly, the message is easy for debugging:
"res operator(): recv : queue result stop"
The left "operator()" is generated by 'RES_DBG() ... __func__'
[1]: https://clang.llvm.org/extra/clang-tidy/checks/bugprone/lambda-function-name.html
Signed-off-by: Haiyue Wang <haiyuewa@163.com>
* server : add Anthropic Messages API support
* remove -@pytest.mark.slow from tool calling/jinja tests
* server : remove unused code and slow/skip on test_anthropic_vision_base64_with_multimodal_model in test_anthropic_api.py
* server : removed redundant n field logic in anthropic_params_from_json
* server : use single error object instead of error_array in streaming response handler for /v1/chat/completions and use unordered_set instead of set in to_json_anthropic_stream()
* server : refactor Anthropic API to use OAI conversion
* make sure basic test always go first
* clean up
* clean up api key check, add test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* webui: add rehype plugin to restore HTML in Markdown table cells
The remark/rehype pipeline neutralizes inline HTML as literal text
(remarkLiteralHtml) so that XML/HTML snippets in LLM responses display
as-is instead of being rendered. This causes <br> and <ul> markup in
table cells to show as plain text.
This plugin traverses the HAST post-conversion, parses whitelisted HTML
patterns (<br>, <ul><li>) from text nodes, and replaces them with actual
HAST element nodes. For lists, adjacent siblings must be combined first
as the AST fragmentation breaks pattern matching.
Strict validation rejects malformed markup, keeping it as raw text.
* chore: update webui build output
* server: split HTTP into its own interface
* move server-http and httplib to its own file
* add the remaining endpoints
* fix exception/error handling
* renaming
* missing header
* fix missing windows header
* fix error responses from http layer
* fix slot save/restore handler
* fix case where only one stream chunk is returned
* add NOMINMAX
* do not call sink.write on empty data
* use safe_json_to_str for SSE
* clean up
* add some comments
* improve usage of next()
* bring back the "server is listening on" message
* more generic handler
* add req.headers
* move the chat template print to init()
* add req.path
* cont : minor
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* webui: add OAI-Compat Harmony tool-call live streaming visualization and persistence in chat UI
- Purely visual and diagnostic change, no effect on model context, prompt
construction, or inference behavior
- Captured assistant tool call payloads during streaming and non-streaming
completions, and persisted them in chat state and storage for downstream use
- Exposed parsed tool call labels beneath the assistant's model info line
with graceful fallback when parsing fails
- Added tool call badges beneath assistant responses that expose JSON tooltips
and copy their payloads when clicked, matching the existing model badge styling
- Added a user-facing setting to toggle tool call visibility to the Developer
settings section directly under the model selector option
* webui: remove scroll listener causing unnecessary layout updates (model selector)
* Update tools/server/webui/src/lib/components/app/chat/ChatMessages/ChatMessageAssistant.svelte
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
* Update tools/server/webui/src/lib/components/app/chat/ChatMessages/ChatMessageAssistant.svelte
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
* chore: npm run format & update webui build output
* chore: update webui build output
---------
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
When compiling llama.cpp in Yocto, it fails QA checks because the generated so files aren't versioned. This applies a version to all generated so files, allowing the package to build without errors.
* feat(memory): Only fail partial erasure of recurrent tail
The recurrent state is always assumed to be the state as of the last update
from the final token in the sequence. When doing a partial erasure, if the
range does not include the final token, the erasure can be considered a
success since any memory used for the sequence prior to the final token
(which is no memory) has been successfully removed.
There is one potential case that this doesn't address which is the pruning
of cache to remove sensitive data from the context. This wouldn't work for
attention cache partial removal (in the middle) either since the KV state
is linearly-dependent and states in later sequence positions would still be
based on the state from the sensitive data, even if that data is no longer
cached, so I don't think this is relevant, but it is worth noting that the
semantics of this change for a partial erasure in the middle of the cache
are essentially "my context is already compressed" and not "all trace of
the removed tokens has been removed."
https://github.com/ggml-org/llama.cpp/issues/16768
Branch: HybridContextShift-16768
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix(main): Check the output of seq_rm for prefix matching
This prefix matching is explicitly attempting to remove the tokens at the
end of the sequence that don't match. This is the operation that can't be
performed on a recurrent cache due to the state being updated in place, so
if this removal fails, we need to clear the whole cache.
https://github.com/ggml-org/llama.cpp/issues/16768
Branch: HybridContextShift-16768
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix(memory): Fix condition for partial erasure failure if p0 > pos
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: compilade <git@compilade.net>
* style: Fix extra parens
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* fix(main.cpp): Set n_matching_session_tokens to 0 on cache clear
https://github.com/ggml-org/llama.cpp/issues/16768
Branch: HybridContextShift-16768
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
---------
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: compilade <git@compilade.net>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
hanishkvc
Adrien Gallouët
Piotr Wilkin (ilintar)
Aleksander Grygier
Andika Wasisto
Pascal
Aldehir Rojas
Xuan-Son Nguyen
Chad Voegele
senhtry
Tarek Dakhran
Haiyue Wang
Igor Smirnov
o7si
Fredrik Hultin
Han Qingzhe
Georgi Gerganov
Ankur Verma
Mike Abbott
Nicolas B. Pierron
Gabe Goodhart