Meta Llama 3.1 8B Instruct
Q4_K - Medium
8.0Bparams
COMPARE ACCELERATORS
200 accelerators tested
Select Accelerators
NVIDIA RTX PRO 6000 Blackwell Workstation Edition
95GB
NVIDIA RTX 6000 Ada Generation
47GB
NVIDIA H100 PCIe
79GB
NVIDIA GeForce RTX 4090
24GB
NVIDIA GeForce RTX 5090
31GB
Meta Llama 3.1 8B Instruct - Q4_K - Medium
LEADERBOARD
PROMPT
8479
tokens/s
GENERATION
138
tokens/s
TTFT
146
ms
LOCALSCORE
2000
GPU / 47GB
PROMPT
6808
tokens/s
GENERATION
121
tokens/s
TTFT
199
ms
LOCALSCORE
1605
GPU / 48GB
PROMPT
5487
tokens/s
GENERATION
51.3
tokens/s
TTFT
252
ms
LOCALSCORE
1038
PROMPT
5483
tokens/s
GENERATION
44.9
tokens/s
TTFT
231
ms
LOCALSCORE
1021
GPU / 16GB
PROMPT
4124
tokens/s
GENERATION
70.7
tokens/s
TTFT
320
ms
LOCALSCORE
967
GPU / 16GB
PROMPT
4461
tokens/s
GENERATION
54.4
tokens/s
TTFT
301
ms
LOCALSCORE
931
GPU / 12GB
PROMPT
3526
tokens/s
GENERATION
56.7
tokens/s
TTFT
376
ms
LOCALSCORE
808
GPU / 20GB
PROMPT
2608
tokens/s
GENERATION
53.5
tokens/s
TTFT
520
ms
LOCALSCORE
645
PROMPT
2457
tokens/s
GENERATION
57.4
tokens/s
TTFT
540
ms
LOCALSCORE
639
GPU / 16GB
PROMPT
1885
tokens/s
GENERATION
52.3
tokens/s
TTFT
699
ms
LOCALSCORE
521
GPU / 20GB
PROMPT
2013
tokens/s
GENERATION
44.6
tokens/s
TTFT
689
ms
LOCALSCORE
507
GPU / 8GB
PROMPT
1460
tokens/s
GENERATION
57.8
tokens/s
TTFT
880
ms
LOCALSCORE
458
GPU / 512GB
PROMPT
1100
tokens/s
GENERATION
62.6
tokens/s
TTFT
1.07
sec
LOCALSCORE
401
PROMPT
1453
tokens/s
GENERATION
40.4
tokens/s
TTFT
920
ms
LOCALSCORE
400
GPU / 256GB
PROMPT
1062
tokens/s
GENERATION
63.3
tokens/s
TTFT
1.10
sec
LOCALSCORE
394
GPU / 8GB
PROMPT
1223
tokens/s
GENERATION
51.3
tokens/s
TTFT
1.04
sec
LOCALSCORE
392
GPU / 16GB
PROMPT
1357
tokens/s
GENERATION
37.5
tokens/s
TTFT
1.00
sec
LOCALSCORE
370
GPU / 6GB
PROMPT
1110
tokens/s
GENERATION
37.3
tokens/s
TTFT
2.54
sec
LOCALSCORE
305
GPU / 8GB
PROMPT
790
tokens/s
GENERATION
27.5
tokens/s
TTFT
1.69
sec
LOCALSCORE
234
GPU / 128GB
PROMPT
534
tokens/s
GENERATION
48.9
tokens/s
TTFT
2.16
sec
LOCALSCORE
230
GPU / 8GB
PROMPT
809
tokens/s
GENERATION
24.3
tokens/s
TTFT
9.09
sec
LOCALSCORE
129
PROMPT
290
tokens/s
GENERATION
4.3
tokens/s
TTFT
8.73
sec
LOCALSCORE
52