Georgi Gerganov
|
ee6d1b3fb4
|
unicode : simplify
|
2024-04-28 18:36:57 +03:00 |
|
Georgi Gerganov
|
e972e6cbf8
|
unicode : clean-up
|
2024-04-28 18:30:37 +03:00 |
|
Georgi Gerganov
|
b97add52a4
|
unicode : category support via std::regex
|
2024-04-28 15:15:57 +03:00 |
|
Georgi Gerganov
|
581c4a0239
|
unicode : try fix windows
|
2024-04-27 18:36:00 +03:00 |
|
Georgi Gerganov
|
91eaa414bf
|
unicode : support \p{N}, \p{L} and \p{P} natively
|
2024-04-27 17:48:38 +03:00 |
|
Georgi Gerganov
|
ce5485aee0
|
unicode : always use std::wregex
|
2024-04-27 17:11:34 +03:00 |
|
Georgi Gerganov
|
a22645c2a7
|
unicode : set bomb
|
2024-04-27 11:48:24 +03:00 |
|
Georgi Gerganov
|
c160818ec0
|
wip
|
2024-04-27 00:28:36 +03:00 |
|
Georgi Gerganov
|
e9891769ff
|
unicode : first try custom implementations
|
2024-04-26 15:09:07 +03:00 |
|
Georgi Gerganov
|
e8c206be61
|
unicode : shot in the dark to fix tests on Windows
|
2024-04-26 14:57:12 +03:00 |
|
Georgi Gerganov
|
06d3e693db
|
unicode : fix? unicode_wstring_to_utf8
|
2024-04-26 12:55:11 +03:00 |
Kazim Abrar Mahi
|
36d983262e
|
Fixed issue with gpt2 regex custom preprocessor
|
2024-04-26 11:43:29 +03:00 |
|
Kazim Abrar Mahi
|
feeaf4f39c
|
Added needed functionality, testing remains
|
2024-04-26 11:43:29 +03:00 |
|
Kazim Abrar Mahi
|
7e308ed212
|
Adding unicode regex function
|
2024-04-26 11:43:29 +03:00 |
|
Kazim Abrar Mahi
|
4056dc5b1e
|
added and refactored unicode_regex_split and related functions
|
2024-04-26 11:43:28 +03:00 |
|
Kazim Abrar Mahi
|
1c924e4b35
|
Resolved issues
|
2024-04-26 11:43:28 +03:00 |
|
Kazim Abrar Mahi
|
54f93eb50b
|
Moved header files
|
2024-04-26 11:43:28 +03:00 |
|
Kazim Abrar Mahi
|
d2cfc2225f
|
Moved regex patterns to unicode.cpp and updated unicode.h
|
2024-04-26 11:43:28 +03:00 |
Jared Van Bortel
|
32c8486e1f
|
wpm : portable unicode tolower (#6305)
Also use C locale for ispunct/isspace, and split unicode-data.cpp from unicode.cpp.
|
2024-03-26 17:46:21 -04:00 |
|
Georgi Gerganov
|
83796e62bc
|
llama : refactor unicode stuff (#5992)
* llama : refactor unicode stuff
ggml-ci
* unicode : names
* make : fix c++ compiler
* unicode : names
* unicode : straighten tables
* zig : fix build
* unicode : put nfd normalization behind API
ggml-ci
* swift : fix build
* unicode : add BOM
* unicode : add <cstdint>
ggml-ci
* unicode : pass as cpts as const ref
|
2024-03-11 17:47:47 +02:00 |