logo
HomeLatest ResultsDownloadAboutBlog
model icon

LLaMA v2

Q4_0

6.7Bparams
COMPARE ACCELERATORS

4 accelerators tested

Select Accelerators

NVIDIA GeForce RTX 5090 Laptop GPU

23GB

NVIDIA GeForce RTX 3080

20GB

Apple M4 Max 12P+4E+40GPU

64GB

Tesla P40

24GB

LLaMA v2 - Q4_0

LEADERBOARD
NVIDIA GeForce RTX 5090 Laptop GPU
GPU / 23GB
PROMPT
5274
tokens/s
GENERATION
125
tokens/s
TTFT
253
ms
LOCALSCORE
1377
NVIDIA GeForce RTX 3080
GPU / 20GB
PROMPT
3128
tokens/s
GENERATION
100
tokens/s
TTFT
411
ms
LOCALSCORE
914
Apple M4 Max 12P+4E+40GPU
GPU / 64GB
PROMPT
692
tokens/s
GENERATION
62.5
tokens/s
TTFT
1.73
sec
LOCALSCORE
293
Tesla P40
GPU / 24GB
PROMPT
833
tokens/s
GENERATION
40.9
tokens/s
TTFT
1.52
sec
LOCALSCORE
282