happyz
/
gemma.cpp
mirror of https://github.com/google/gemma.cpp.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
244 Commits 20 Branches 5 Tags 35 MiB
Commit Graph

4 Commits

Author SHA1 Message Date
Copybara-Service befe9fb07e Merge pull request #167 from szabadka:gemma2 
PiperOrigin-RevId: 629325219
 2024-04-30 01:00:37 -07:00
Zoltan Szabadka 27117cc39f Simplify threading: remove the use of inner_pool. 
We only used inner_pool in the prefill FFW function, and there we
can achieve sufficient parallelism on the rows of the matrix-vector
multiplications.

Benchmark results on a 1600-token summarization task:

```
               Prefill speed
Num threads    BEFORE         AFTER
4               9.24 t/s       9.76 t/s
18             31.41 t/s      31.16 t/s
32             31.41 t/s      45.13 t/s
64             31.03 t/s      57.85 t/s
```
 2024-04-29 16:07:30 +00:00
Andrey Mikhaylov 4ef3da733a Fixed minor things and added comments. 2024-04-12 15:39:16 +00:00
Andrey Mikhaylov 03284d752e Added layers output functionality to gemma and a binary debug_output to save the outputs to a json file. 2024-04-12 15:39:16 +00:00