happyz
/
llama.cpp
mirror of https://github.com/ggerganov/llama.cpp.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Fixed issue with gpt2 regex custom preprocessor

This commit is contained in:
Kazim Abrar Mahi 2024-04-17 07:40:40 +06:00 committed by Georgi Gerganov
parent 753580360b
commit 36d983262e
No known key found for this signature in database
GPG Key ID: 449E073F9DC10735
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
unicode.cpp
View File

@ -357,6 +357,7 @@ static std::vector<size_t> unicode_gpt2_regex_preprocess(const std::wstring & wt
token += utf_char; token += utf_char;
} }
} }
start += offset;
} }
return bpe_offsets; return bpe_offsets;