Commit Graph

316 Commits

Author SHA1 Message Date
HanishKVC 8fe8231313 ChatON:SubPartsAwareTokenizePath: Allow extract subparts testing 2024-05-08 19:51:57 +05:30
HanishKVC a49697b488 ChatON: Keep compiler happy simbly 2024-05-08 19:22:46 +05:30
HanishKVC 868ab608f0 ChatON: Add forceParseSpecial flag to subparts aware tokenizing 2024-05-08 18:42:22 +05:30
HanishKVC b6da7d9c9d ChatON: tokenize keeping in mind the taggedMessage subparts
Initial go
2024-05-08 18:38:07 +05:30
HanishKVC 8dfa31bb91 ChatON: Make c-api wrappers a bit robust incl some cross checks
If the tagged message will be of 0 length, ensure that the passed
dest char* array, has null inserted appropriately.

Check that user has passed a non-null pNumParts.

Dont hard code int32_t size, pick using sizeof
2024-05-08 17:05:45 +05:30
HanishKVC 76791bad63 ChatON:Fix partsLengths to int32_t type, instead of int
so that the size of the elements is explicit and fixed, so that
it is inturn in sync with the fixed int size specified wrt the
c-api, even with any c compilers with different idea about int.

avoid some ununsed vars, need to update compile flags later to
enable corresponding warnings.
2024-05-07 12:40:49 +05:30
HanishKVC b3a56545d6 ChatON:Reposition alertAssistantAtEnd flag for consistency 2024-05-07 11:49:43 +05:30
HanishKVC 0852f3b7ec ChatON:ExCApi: Rename for consistency 2024-05-07 11:46:40 +05:30
HanishKVC 43a3a91b03 ChatON: Cleanup/Refine initial go at tmpl_apply_ex_capi 2024-05-07 11:44:25 +05:30
HanishKVC 7c288d3dfc ChatON: Rename to partstypes for consistency 2024-05-07 11:32:20 +05:30
HanishKVC 04b4a15177 ChatON: Initial go at chat-template-apply c-api with parts info 2024-05-07 11:08:47 +05:30
HanishKVC f6a86cd209 ChatON: Update the Note a bit 2024-05-07 10:29:16 +05:30
HanishKVC 989c6c4125 SimpCfg: Cleanup the Note a bit to avoid some ambiguities 2024-05-06 11:27:56 +05:30
HanishKVC 344c068d7b SimpCfg:MultiPart keys wrt get_vector
With this and past few commits, now there is simple yet sufficient
support to help move multi-level-hierarchy config files into the
SimpCfg's simple physically 1-level, but if reqd logically multi
level hierarchy flow.

B4 this series of commits also one could have still achieved this,
but there would have been bit more effort needed.
2024-05-06 11:27:56 +05:30
HanishKVC 19d3c88e8a SimpCfg:MultiPart keys wrt get_value etal 2024-05-06 11:27:56 +05:30
HanishKVC 623d0b60da SimpCfg: General MultiPart support, KeyParts not Key wrt SetValue 2024-05-06 11:27:56 +05:30
HanishKVC c6ecd9316e SimpCfg: Use to_str instead of using stringstream directly 2024-05-06 11:27:56 +05:30
HanishKVC 2b14bcaddb SimpCfg:ChatON: add by Humans for All note 2024-05-06 11:27:56 +05:30
HanishKVC 20e5b383c5 SimpCfg:Trim DumpHexString only if SC_DEBUG_VERBOSE 2024-05-06 11:27:56 +05:30
HanishKVC f53c19baac SimpCfg: Update the notes wrt tolower and add test code 2024-05-06 11:27:56 +05:30
HanishKVC 3287fdba28 SimpCfg:Fix/cleanup trim related test samples and flow
Use the commonality between Indian languages to show mixup issue
with the simple minded trim_dump logic and how trim_oversmart
could potentially avoid that.

Given that I am using valid strings to show the pitfalls of fixed
native char size driven logic, so no need to keep the dump and
oversmart flows seperate, so merge into a common loop.
2024-05-06 11:27:56 +05:30
HanishKVC 33619a3b92 SimpCfg: Templatize str_lower 2024-05-06 11:27:56 +05:30
HanishKVC 32ba195a83 SimpCfg: Templatize str_trim_single
Also use NativeCharSize and MultiNativeCharSize wording to make
the note more generic
2024-05-06 11:27:56 +05:30
HanishKVC 5b8bf849c0 SimpCfg: Fixed & ~Variable Length to Native & MultiNativeCharSize
So as to make the notes, more generic.
2024-05-06 11:27:56 +05:30
HanishKVC d030a26f3c SimpCfg:Update TrimOverSmart use templated TrimDumb after wstrconv 2024-05-06 11:27:56 +05:30
HanishKVC 97ac443bba SimpCfg:Cleanup, updated notes, templated code
Update the notes to match the templated flow now and some of the
nitty gritties involved.

Update DumpHexString to be templated.

Split check nonenglish flow wrt trim dumb and oversmart testing,
so that things with work with one, but not the other can be
differentiated in the flow.
2024-05-06 11:27:56 +05:30
HanishKVC bf111a83f1 SimpCfg:TemplatedDumbTrim; Test dumb and oversmart trim logics 2024-05-06 11:27:56 +05:30
HanishKVC 554b00f027 SimpCfg: Add some missing const refs 2024-05-06 11:27:56 +05:30
HanishKVC cae0fff715 SimpCfg: Update notes; Try add a better trimming logic 2024-05-06 11:27:56 +05:30
HanishKVC d1156cc055 SimpCfg: As locale manipulation reqd for better processing 2024-05-06 11:27:56 +05:30
HanishKVC 2325764180 SimpCfg:CheckStrings: Switch Mbs2Wcs to multithread safe calls 2024-05-06 11:27:56 +05:30
HanishKVC 23acf07bb2 SimpCfg:CheckStrings: Cleanup wstring flow to needed parts 2024-05-06 11:27:56 +05:30
HanishKVC 2cda78f1ad SimpCfg:CheckStrings: WString2String finally
The constructor method doesnt convert wstring to string, when it
involves non-english chars which will encode to multibyte chars
in utf8. even thou it does work for the already utf8 u8string.

wcstombs doesnt seem to work for non english chars, when the
locale is set to the default c, need to change to something like
en_US.UTF-8, to allow it to do the conversion properly.
2024-05-06 11:27:56 +05:30
HanishKVC 7607dbc8c7 SimpCfg:CheckStrings: Try fixup wstring handling 2024-05-06 11:27:56 +05:30
HanishKVC 1a618a42f8 SimpCfg: Update the func notes with alert 2024-05-06 11:27:56 +05:30
HanishKVC 66d6fa62b7 SimpCfg: C++ and strings is a mess even after decades
Seperate out the checks wrt different string types.

Add a wstring_basic, which verifies that wstring iterator handles
non english chars propery or atleast better.
2024-05-06 11:27:56 +05:30
HanishKVC 3ad5cec47e SimpCfg:CheckStrings:MacOS, wstring and wcout
Without using imbue, I couldnt get non-english wstrings to print
on mac. Need to check on linux also.

Also avoid the uint8_t typecasting, given that wchar isnt 8bit
2024-05-06 11:27:56 +05:30
HanishKVC a448fec486 SimpCfg:CheckString: organise and probe - p3 wstring
wcouts' involving 2nd wstring with non english char in it not
showing up?
2024-05-06 11:27:56 +05:30
HanishKVC 691d0d43b5 SimpCfg:CheckStrings: Organise and Probe - P2 - std::u8string 2024-05-06 11:27:56 +05:30
HanishKVC 713520caac SimpCfg:CheckStrings: Organise and Probe - p1 std::string 2024-05-06 11:27:56 +05:30
HanishKVC 56f19c7a68 SimpCfg: Test c++ string handling 2024-05-06 11:27:56 +05:30
HanishKVC 86e776c857 SimpCfg: Rename to get_vector, add some test code 2024-05-06 11:27:56 +05:30
HanishKVC 1e1f54ec1d SimpCfg:GetArray flesh out, helpers to convert to string
Good: Forgotten love, All the light we cant see, laapataa ladies
2024-05-06 11:27:56 +05:30
HanishKVC 561f50930e SimpCfg: initial go at adding support for spreadout arrays
By having a seperate entry for each element of the array, the
existing logic itself can be repurposed with minimal additions.
2024-05-06 11:27:56 +05:30
HanishKVC 8fdc80533f SimpCfg:Cleanup:Cmdline Arg, GetValueCallerLogging, StringCmp
Bring the direct string comparison also back, as the issue with
cmp was more to do with string.transform.
2024-05-06 11:27:56 +05:30
HanishKVC 08b9711d68 SimpCfg:Remove dbug logs wrt str_tolower and set_bool
Also the warning wrt is it string, now also logs the line number,
group and key, to help user identify the line better.

Misc: pass time last week Another life, Anchakkallakokkan, Deadloch
2024-05-06 11:27:56 +05:30
HanishKVC 0e0d7da18f SimpCfg:Found issue with str_tolower, transform doesnt resize dst 2024-05-06 11:27:56 +05:30
HanishKVC eb56517f20 SimpCfg:Bools:Make lowercase b4 checking true/false for bool path
TODO: string check wrt true/false, doesnt seem to be working after
str_tolower was introduced. I seem to be doing some silly mistake
not able to make out, moving in and out of sleep, need to check
tomorrow.

string == string-literal failed
string == string-view failed
string.compare(string-literal) failed

Bit strange
2024-05-06 11:27:56 +05:30
HanishKVC ef5a2cf391 SimpCfg:Dbug why bool is not setting properly 2024-05-06 11:27:56 +05:30
HanishKVC 5aa1072aac SimpCfg: Move dump into its own func, Avoid KV iter wrt Get 2024-05-06 11:27:56 +05:30