4 Commits

Author	SHA1	Message	Date
Kazim Abrar Mahi	54f93eb50b	Moved header files	2024-04-26 11:43:28 +03:00
Kazim Abrar Mahi	d2cfc2225f	Moved regex patterns to unicode.cpp and updated unicode.h	2024-04-26 11:43:28 +03:00
Jared Van Bortel	32c8486e1f	wpm : portable unicode tolower (#6305) ... Also use C locale for ispunct/isspace, and split unicode-data.cpp from unicode.cpp.	2024-03-26 17:46:21 -04:00
Georgi Gerganov	83796e62bc	llama : refactor unicode stuff (#5992) ... * llama : refactor unicode stuff ggml-ci * unicode : names * make : fix c++ compiler * unicode : names * unicode : straighten tables * zig : fix build * unicode : put nfd normalization behind API ggml-ci * swift : fix build * unicode : add BOM * unicode : add <cstdint> ggml-ci * unicode : pass as cpts as const ref	2024-03-11 17:47:47 +02:00