happyz
/
gemma.cpp
mirror of https://github.com/google/gemma.cpp.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
487 Commits 18 Branches 5 Tags 35 MiB
Commit Graph

2 Commits

Author SHA1 Message Date
Jan Wassenberg aaf51898b6 Major revamp #2 of Prefill: fix token order, parallel for multi-query 
- Allocate only the required KV caches and activation batch size
- Add flags for batch sizes
- Const-correct interface: Span of const int.
- Also clean up the KVCache arg to a span.
- Move kPrefillBatchSize into RuntimeConfig and remove related global constants.

PiperOrigin-RevId: 655893197
 2024-07-25 03:28:55 -07:00
Jan Wassenberg 09a7e75ead Prep for sharding gemma.cc: split into kv_cache, tokenizer. 
Move activations.h to backprop/ to make space for another activations.h.

PiperOrigin-RevId: 648744500
 2024-07-02 09:31:06 -07:00