logo
HomeLatest ResultsDownloadAboutBlog
model icon

LLaMA v2

Q4_0

6.7Bparams
COMPARE ACCELERATORS

3 accelerators tested

Select Accelerators

NVIDIA GeForce RTX 3080

20GB

Apple M4 Max 12P+4E+40GPU

64GB

Tesla P40

24GB

LLaMA v2 - Q4_0

LEADERBOARD
NVIDIA GeForce RTX 3080
GPU / 20GB
PROMPT
3128
tokens/s
GENERATION
100
tokens/s
TTFT
411
ms
LOCALSCORE
914
Apple M4 Max 12P+4E+40GPU
GPU / 64GB
PROMPT
692
tokens/s
GENERATION
62.5
tokens/s
TTFT
1.73
sec
LOCALSCORE
293
Tesla P40
GPU / 24GB
PROMPT
833
tokens/s
GENERATION
40.9
tokens/s
TTFT
1.52
sec
LOCALSCORE
282