3 Commits

Author	SHA1	Message	Date
Kerfuffle	28a2e6e7d4	tokenize example: Respect normal add BOS token behavior (#4126) ... Allow building with Makefile	2023-11-18 14:48:17 -07:00
Georgi Gerganov	5ad387e994	tokenize : fix trailing whitespace	2023-11-17 18:01:38 +02:00
zakkor	2fa02b4b3d	examples : add tokenize (#4039)	2023-11-17 17:36:44 +02:00