happyz
/
gemma.cpp
mirror of https://github.com/google/gemma.cpp.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
415 Commits 20 Branches 5 Tags 35 MiB
Commit Graph

7 Commits

Author SHA1 Message Date
Jan Wassenberg 22995c699d Simplify pos handling, auto-increment output arg 
- no longer multiply by num_queries
- remove unused interleaved prompts
- Rename to Queries*
- Rename batch_start/interleaved_pos/pos to queries_pos

PiperOrigin-RevId: 663331823
 2024-08-15 09:25:26 -07:00
RangerUFO 730b6bfc94 Implement `start_pos` per query for batch interface 2024-08-12 18:50:23 +02:00
The gemma.cpp Authors 27258b03e6 Improve performance logging 
PiperOrigin-RevId: 660534330
 2024-08-07 14:15:43 -07:00
Jan Wassenberg 5e433e774a 1.1x prefill speedup, revamp threading in preparation for hierarchical parallelism. 
Limit thread counts to detected. Add max_clusters arg.
Update detection logic to check for smt0 - previously we pinned to some siblings.

PiperOrigin-RevId: 659755311
 2024-08-05 18:50:09 -07:00
Jan Wassenberg aaf51898b6 Major revamp #2 of Prefill: fix token order, parallel for multi-query 
- Allocate only the required KV caches and activation batch size
- Add flags for batch sizes
- Const-correct interface: Span of const int.
- Also clean up the KVCache arg to a span.
- Move kPrefillBatchSize into RuntimeConfig and remove related global constants.

PiperOrigin-RevId: 655893197
 2024-07-25 03:28:55 -07:00
Jan Wassenberg 12016d31c3 Major Prefill/Generate cleanup, 1.3x Prefill speedup 
This fixes TTFT, which was not including prefill.

PiperOrigin-RevId: 653690626
 2024-07-18 11:16:46 -07:00
Jan Wassenberg cbb67b4ee0 Move benchmark_helper to evals/, weights_raw to compression/. 
PiperOrigin-RevId: 650155983
 2024-07-08 01:13:23 -07:00
Renamed from gemma/benchmark_helper.cc (Browse further)