Commit Graph

3 Commits

Author SHA1 Message Date
Jared Van Bortel b460e7f5b4 wpm : portable unicode tolower 2024-03-25 16:12:33 -04:00
Jared Van Bortel e5ddf2fcdd llama : split unicodedata.cpp from unicode.cpp 2024-03-25 16:12:32 -04:00
Georgi Gerganov 83796e62bc
llama : refactor unicode stuff (#5992)
* llama : refactor unicode stuff

ggml-ci

* unicode : names

* make : fix c++ compiler

* unicode : names

* unicode : straighten tables

* zig : fix build

* unicode : put nfd normalization behind API

ggml-ci

* swift : fix build

* unicode : add BOM

* unicode : add <cstdint>

ggml-ci

* unicode : pass as cpts as const ref
2024-03-11 17:47:47 +02:00