gemma.cpp

409 Commits 18 Branches 5 Tags 35 MiB

Author	SHA1	Message	Date
Jan Wassenberg	5e433e774a	1.1x prefill speedup, revamp threading in preparation for hierarchical parallelism. Limit thread counts to detected. Add max_clusters arg. Update detection logic to check for smt0 - previously we pinned to some siblings. PiperOrigin-RevId: 659755311	2024-08-05 18:50:09 -07:00
Jan Wassenberg	aaf51898b6	Major revamp #2 of Prefill: fix token order, parallel for multi-query - Allocate only the required KV caches and activation batch size - Add flags for batch sizes - Const-correct interface: Span of const int. - Also clean up the KVCache arg to a span. - Move kPrefillBatchSize into RuntimeConfig and remove related global constants. PiperOrigin-RevId: 655893197	2024-07-25 03:28:55 -07:00
Jan Wassenberg	cbb67b4ee0	Move benchmark_helper to evals/, weights_raw to compression/. PiperOrigin-RevId: 650155983	2024-07-08 01:13:23 -07:00

Author

SHA1

Message

Date

Jan Wassenberg

5e433e774a

1.1x prefill speedup, revamp threading in preparation for hierarchical parallelism.

Limit thread counts to detected. Add max_clusters arg.
Update detection logic to check for smt0 - previously we pinned to some siblings.

PiperOrigin-RevId: 659755311

2024-08-05 18:50:09 -07:00

Jan Wassenberg

aaf51898b6

Major revamp #2 of Prefill: fix token order, parallel for multi-query

- Allocate only the required KV caches and activation batch size
- Add flags for batch sizes
- Const-correct interface: Span of const int.
- Also clean up the KVCache arg to a span.
- Move kPrefillBatchSize into RuntimeConfig and remove related global constants.

PiperOrigin-RevId: 655893197

2024-07-25 03:28:55 -07:00

Jan Wassenberg

cbb67b4ee0

Move benchmark_helper to evals/, weights_raw to compression/.

PiperOrigin-RevId: 650155983

2024-07-08 01:13:23 -07:00

Renamed from gemma/benchmark_helper.h (Browse further)

3 Commits