gemma.cpp

246 Commits 18 Branches 5 Tags 36 MiB

Author	SHA1	Message	Date
Apoorv Reddy	f1eab987d8	Store tokens/sec in auxiliary struct TimingInfo. PiperOrigin-RevId: 633108908	2024-05-13 00:04:19 -07:00
Zoltan Szabadka	27117cc39f	Simplify threading: remove the use of inner_pool. We only used inner_pool in the prefill FFW function, and there we can achieve sufficient parallelism on the rows of the matrix-vector multiplications. Benchmark results on a 1600-token summarization task: ``` Prefill speed Num threads BEFORE AFTER 4 9.24 t/s 9.76 t/s 18 31.41 t/s 31.16 t/s 32 31.41 t/s 45.13 t/s 64 31.03 t/s 57.85 t/s ```	2024-04-29 16:07:30 +00:00
Jan Wassenberg	a982ec1287	Move code to gemma/ so we can remove error-prone copybara: comments. Also fix includes and Lint warnings. PiperOrigin-RevId: 623127487	2024-04-09 04:45:42 -07:00

Author

SHA1

Message

Date

Apoorv Reddy

f1eab987d8

Store tokens/sec in auxiliary struct TimingInfo.

PiperOrigin-RevId: 633108908

2024-05-13 00:04:19 -07:00

Zoltan Szabadka

27117cc39f

Simplify threading: remove the use of inner_pool.

We only used inner_pool in the prefill FFW function, and there we
can achieve sufficient parallelism on the rows of the matrix-vector
multiplications.

Benchmark results on a 1600-token summarization task:

```
               Prefill speed
Num threads    BEFORE         AFTER
4               9.24 t/s       9.76 t/s
18             31.41 t/s      31.16 t/s
32             31.41 t/s      45.13 t/s
64             31.03 t/s      57.85 t/s
```

2024-04-29 16:07:30 +00:00

Jan Wassenberg

a982ec1287

Move code to gemma/ so we can remove error-prone copybara: comments.

Also fix includes and Lint warnings.

PiperOrigin-RevId: 623127487

2024-04-09 04:45:42 -07:00

Renamed from benchmark.cc (Browse further)

3 Commits