Rename all the log messages to have GKV and not SC.
The log messages in get_vector made conditional to GKV_DEBUG, this
was missed out earlier in simpcfg itself.
This should pave way for having a default chat templates dataset
in the code, without needing to load it from a config file, if
one doesnt want to.
TODO: allow for loading config from json into simpcfg, so that
a program which uses llama.cpp can decide, whether it is ok with
what is already there in the internal dataset, or allow for loading
template info at runtime using the simpcfg's simple text file or
additionally include the json code to load template info at runtime
from json file.
It appears like std::format is not supported in older g++/lib still
in wide use like current debian stable, so avoiding same wrt direct
library use.
Allow for empty VAARGS
NOTE: However test program mode of the same uses cout and format
The llama.cpp grammar parser had a bug where forgetting to add a closing
quotation mark to strings would cause parsing to crash. Anyone running a
server on a public endpoint is advised to upgrade. To reproduce this bug
./llamafile -m foo.gguf -p bar --grammar 'root::="'
Credit for discovering and reporting this issue goes to Eclypsium
Security Researcher Richard Johnson <Richard.johnson@eclypsium.com>.
Have merged master branch has of 20240510IST12XY with chaton_v3
branch.
As part of same had to update the flow in examples/main/main.cpp
wrt conversion related commit in master branch and my chaton related
commits in this branch.
If the tagged message will be of 0 length, ensure that the passed
dest char* array, has null inserted appropriately.
Check that user has passed a non-null pNumParts.
Dont hard code int32_t size, pick using sizeof
so that the size of the elements is explicit and fixed, so that
it is inturn in sync with the fixed int size specified wrt the
c-api, even with any c compilers with different idea about int.
avoid some ununsed vars, need to update compile flags later to
enable corresponding warnings.
With this and past few commits, now there is simple yet sufficient
support to help move multi-level-hierarchy config files into the
SimpCfg's simple physically 1-level, but if reqd logically multi
level hierarchy flow.
B4 this series of commits also one could have still achieved this,
but there would have been bit more effort needed.
Use the commonality between Indian languages to show mixup issue
with the simple minded trim_dump logic and how trim_oversmart
could potentially avoid that.
Given that I am using valid strings to show the pitfalls of fixed
native char size driven logic, so no need to keep the dump and
oversmart flows seperate, so merge into a common loop.
Update the notes to match the templated flow now and some of the
nitty gritties involved.
Update DumpHexString to be templated.
Split check nonenglish flow wrt trim dumb and oversmart testing,
so that things with work with one, but not the other can be
differentiated in the flow.
The constructor method doesnt convert wstring to string, when it
involves non-english chars which will encode to multibyte chars
in utf8. even thou it does work for the already utf8 u8string.
wcstombs doesnt seem to work for non english chars, when the
locale is set to the default c, need to change to something like
en_US.UTF-8, to allow it to do the conversion properly.
Seperate out the checks wrt different string types.
Add a wstring_basic, which verifies that wstring iterator handles
non english chars propery or atleast better.
Without using imbue, I couldnt get non-english wstrings to print
on mac. Need to check on linux also.
Also avoid the uint8_t typecasting, given that wchar isnt 8bit
Also the warning wrt is it string, now also logs the line number,
group and key, to help user identify the line better.
Misc: pass time last week Another life, Anchakkallakokkan, Deadloch
TODO: string check wrt true/false, doesnt seem to be working after
str_tolower was introduced. I seem to be doing some silly mistake
not able to make out, moving in and out of sleep, need to check
tomorrow.
string == string-literal failed
string == string-view failed
string.compare(string-literal) failed
Bit strange
test-chat-template-chaton now tries to check if meta-ok is ok wrt
the template-id being looked into.
Log template-id info also, where it was previously missed out.
Warn if something not starting with double quote is being treated
as a string.
Show some examples of invalid floating point values wrt this
logics floating point determination code
so that one can update the value-item's content, without needing
to explicitly update/store the value-item back into map after the
content has been updated.
This should make these setting operations/helpers more efficient.
As c doesnt have the concept of pass by reference, and inturn the
existing c api uses pointers wrt llama chat message structure, so
switching to same wrt chat_tmpl_apply logics.
Also fix a oversight in previous commit and add the remaining logic.
Initial skeletons
Update existing logics to help with same. Also the inbetween helper
was having a bad signature wrt returning status and data, thats also
fixed.
HanishKVC
Justine Tunney
Johannes Gäßler
Dawid Potocki