Meta Llama 3.1 8B Instruct
Q4_K - Medium
8.0Bparams
COMPARE ACCELERATORS
266 accelerators tested
Select Accelerators
NVIDIA RTX PRO 6000 Blackwell Workstation Edition
95GB
NVIDIA GeForce RTX 4090
23GB
NVIDIA RTX 6000 Ada Generation
47GB
NVIDIA GeForce RTX 4090
24GB
NVIDIA H100 PCIe
79GB
Meta Llama 3.1 8B Instruct - Q4_K - Medium
LEADERBOARD
PROMPT
8479
tokens/s
GENERATION
138
tokens/s
TTFT
146
ms
LOCALSCORE
2000
GPU / 47GB
PROMPT
6808
tokens/s
GENERATION
121
tokens/s
TTFT
199
ms
LOCALSCORE
1605
PROMPT
5544
tokens/s
GENERATION
46.2
tokens/s
TTFT
228
ms
LOCALSCORE
1039
GPU / 48GB
PROMPT
5487
tokens/s
GENERATION
51.3
tokens/s
TTFT
252
ms
LOCALSCORE
1038
GPU / 16GB
PROMPT
4124
tokens/s
GENERATION
70.7
tokens/s
TTFT
320
ms
LOCALSCORE
967
GPU / 16GB
PROMPT
4461
tokens/s
GENERATION
54.4
tokens/s
TTFT
301
ms
LOCALSCORE
931
GPU / 12GB
PROMPT
3526
tokens/s
GENERATION
56.7
tokens/s
TTFT
376
ms
LOCALSCORE
808
GPU / 20GB
PROMPT
2608
tokens/s
GENERATION
53.5
tokens/s
TTFT
520
ms
LOCALSCORE
645
PROMPT
2445
tokens/s
GENERATION
41.8
tokens/s
TTFT
543
ms
LOCALSCORE
564
GPU / 8GB
PROMPT
2052
tokens/s
GENERATION
53.9
tokens/s
TTFT
635
ms
LOCALSCORE
558
GPU / 16GB
PROMPT
1885
tokens/s
GENERATION
52.3
tokens/s
TTFT
699
ms
LOCALSCORE
521
GPU / 8GB
PROMPT
2131
tokens/s
GENERATION
40.9
tokens/s
TTFT
648
ms
LOCALSCORE
512
GPU / 8GB
PROMPT
1693
tokens/s
GENERATION
59.8
tokens/s
TTFT
756
ms
LOCALSCORE
512
GPU / 20GB
PROMPT
2004
tokens/s
GENERATION
44.4
tokens/s
TTFT
690
ms
LOCALSCORE
505
GPU / 8GB
PROMPT
1460
tokens/s
GENERATION
57.8
tokens/s
TTFT
880
ms
LOCALSCORE
458
GPU / 512GB
PROMPT
1109
tokens/s
GENERATION
62.7
tokens/s
TTFT
1.06
sec
LOCALSCORE
403
PROMPT
1453
tokens/s
GENERATION
40.4
tokens/s
TTFT
920
ms
LOCALSCORE
400
GPU / 256GB
PROMPT
1062
tokens/s
GENERATION
63.3
tokens/s
TTFT
1.10
sec
LOCALSCORE
394
GPU / 8GB
PROMPT
1223
tokens/s
GENERATION
51.3
tokens/s
TTFT
1.04
sec
LOCALSCORE
392
GPU / 16GB
PROMPT
1357
tokens/s
GENERATION
37.5
tokens/s
TTFT
1.00
sec
LOCALSCORE
370
GPU / 6GB
PROMPT
1253
tokens/s
GENERATION
40.8
tokens/s
TTFT
1.82
sec
LOCALSCORE
353
GPU / 64GB
PROMPT
703
tokens/s
GENERATION
59.5
tokens/s
TTFT
1.64
sec
LOCALSCORE
294
GPU / 6GB
PROMPT
1185
tokens/s
GENERATION
31.0
tokens/s
TTFT
1.46
sec
LOCALSCORE
293
GPU / 128GB
PROMPT
668
tokens/s
GENERATION
54.3
tokens/s
TTFT
1.69
sec
LOCALSCORE
278
PROMPT
899
tokens/s
GENERATION
26.9
tokens/s
TTFT
1.49
sec
LOCALSCORE
253
GPU / 8GB
PROMPT
790
tokens/s
GENERATION
27.5
tokens/s
TTFT
1.69
sec
LOCALSCORE
234
GPU / 128GB
PROMPT
534
tokens/s
GENERATION
48.9
tokens/s
TTFT
2.16
sec
LOCALSCORE
230
PROMPT
906
tokens/s
GENERATION
19.8
tokens/s
TTFT
1.65
sec
LOCALSCORE
225
GPU / 8GB
PROMPT
809
tokens/s
GENERATION
24.3
tokens/s
TTFT
9.09
sec
LOCALSCORE
129
GPU / 4GB
PROMPT
280
tokens/s
GENERATION
4.8
tokens/s
TTFT
7.47
sec
LOCALSCORE
56
PROMPT
290
tokens/s
GENERATION
4.3
tokens/s
TTFT
8.73
sec
LOCALSCORE
52
GPU / 6GB
PROMPT
133
tokens/s
GENERATION
10.0
tokens/s
TTFT
10.58
sec
LOCALSCORE
50