logo
HomeLatest ResultsDownloadAboutBlog
model icon

Meta Llama 3.1 8B Instruct

Q4_K - Medium

8.0Bparams
COMPARE ACCELERATORS

200 accelerators tested

Select Accelerators

NVIDIA RTX PRO 6000 Blackwell Workstation Edition

95GB

NVIDIA RTX 6000 Ada Generation

47GB

NVIDIA H100 PCIe

79GB

NVIDIA GeForce RTX 4090

24GB

NVIDIA GeForce RTX 5090

31GB

Meta Llama 3.1 8B Instruct - Q4_K - Medium

LEADERBOARD
NVIDIA RTX PRO 6000 Blackwell Workstation Edition
GPU / 95GB
PROMPT
8479
tokens/s
GENERATION
138
tokens/s
TTFT
146
ms
LOCALSCORE
2000
NVIDIA RTX 6000 Ada Generation
GPU / 47GB
PROMPT
6808
tokens/s
GENERATION
121
tokens/s
TTFT
199
ms
LOCALSCORE
1605
NVIDIA H100 PCIe
GPU / 79GB
PROMPT
5202
tokens/s
GENERATION
120
tokens/s
TTFT
236
ms
LOCALSCORE
1382
NVIDIA GeForce RTX 4090
GPU / 24GB
PROMPT
6202
tokens/s
GENERATION
78.7
tokens/s
TTFT
224
ms
LOCALSCORE
1306
NVIDIA GeForce RTX 5090
GPU / 31GB
PROMPT
6477
tokens/s
GENERATION
65.1
tokens/s
TTFT
217
ms
LOCALSCORE
1260
NVIDIA GeForce RTX 4080
GPU / 16GB
PROMPT
4792
tokens/s
GENERATION
88.3
tokens/s
TTFT
279
ms
LOCALSCORE
1149
NVIDIA GeForce RTX 3090 Ti
GPU / 24GB
PROMPT
4024
tokens/s
GENERATION
110
tokens/s
TTFT
320
ms
LOCALSCORE
1113
NVIDIA A100-SXM4-80GB
GPU / 79GB
PROMPT
3571
tokens/s
GENERATION
110
tokens/s
TTFT
336
ms
LOCALSCORE
1054
NVIDIA GeForce RTX 3080 Ti
GPU / 12GB
PROMPT
3739
tokens/s
GENERATION
104
tokens/s
TTFT
343
ms
LOCALSCORE
1042
NVIDIA RTX 6000 Ada Generation
GPU / 48GB
PROMPT
5487
tokens/s
GENERATION
51.3
tokens/s
TTFT
252
ms
LOCALSCORE
1038
NVIDIA GeForce RTX 5070 Ti
GPU / 15GB
PROMPT
4323
tokens/s
GENERATION
79.8
tokens/s
TTFT
315
ms
LOCALSCORE
1031
NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
GPU / 95GB
PROMPT
5483
tokens/s
GENERATION
44.9
tokens/s
TTFT
231
ms
LOCALSCORE
1021
NVIDIA GeForce RTX 3090
GPU / 24GB
PROMPT
3692
tokens/s
GENERATION
98.0
tokens/s
TTFT
347
ms
LOCALSCORE
1013
NVIDIA L40S
GPU / 45GB
PROMPT
5328
tokens/s
GENERATION
45.9
tokens/s
TTFT
262
ms
LOCALSCORE
977
NVIDIA GeForce RTX 4070 Ti SUPER
GPU / 16GB
PROMPT
4124
tokens/s
GENERATION
70.7
tokens/s
TTFT
320
ms
LOCALSCORE
967
NVIDIA L40
GPU / 44GB
PROMPT
5065
tokens/s
GENERATION
45.1
tokens/s
TTFT
274
ms
LOCALSCORE
941
NVIDIA RTX A6000
GPU / 48GB
PROMPT
3447
tokens/s
GENERATION
90.5
tokens/s
TTFT
378
ms
LOCALSCORE
938
NVIDIA GeForce RTX 4080 SUPER
GPU / 16GB
PROMPT
4461
tokens/s
GENERATION
54.4
tokens/s
TTFT
301
ms
LOCALSCORE
931
NVIDIA H100 80GB HBM3
GPU / 79GB
PROMPT
4687
tokens/s
GENERATION
42.4
tokens/s
TTFT
263
ms
LOCALSCORE
911
NVIDIA GeForce RTX 5080
GPU / 15GB
PROMPT
4428
tokens/s
GENERATION
44.9
tokens/s
TTFT
289
ms
LOCALSCORE
883
NVIDIA A40
GPU / 44GB
PROMPT
3270
tokens/s
GENERATION
80.6
tokens/s
TTFT
401
ms
LOCALSCORE
870
NVIDIA GeForce RTX 4070 Ti
GPU / 12GB
PROMPT
3692
tokens/s
GENERATION
60.0
tokens/s
TTFT
371
ms
LOCALSCORE
839
NVIDIA GeForce RTX 4070
GPU / 12GB
PROMPT
3192
tokens/s
GENERATION
76.3
tokens/s
TTFT
415
ms
LOCALSCORE
838
NVIDIA RTX A5000
GPU / 24GB
PROMPT
2891
tokens/s
GENERATION
89.1
tokens/s
TTFT
446
ms
LOCALSCORE
832
NVIDIA H100 NVL
GPU / 93GB
PROMPT
4299
tokens/s
GENERATION
37.1
tokens/s
TTFT
289
ms
LOCALSCORE
820
NVIDIA GeForce RTX 3080
GPU / 10GB
PROMPT
2949
tokens/s
GENERATION
78.3
tokens/s
TTFT
431
ms
LOCALSCORE
812
NVIDIA GeForce RTX 4070 SUPER
GPU / 12GB
PROMPT
3526
tokens/s
GENERATION
56.7
tokens/s
TTFT
376
ms
LOCALSCORE
808
NVIDIA GeForce RTX 3070 Ti
GPU / 8GB
PROMPT
2509
tokens/s
GENERATION
83.2
tokens/s
TTFT
520
ms
LOCALSCORE
738
NVIDIA RTX A4500
GPU / 20GB
PROMPT
2484
tokens/s
GENERATION
77.6
tokens/s
TTFT
522
ms
LOCALSCORE
718
NVIDIA GeForce RTX 5060 Ti
GPU / 15GB
PROMPT
2485
tokens/s
GENERATION
61.3
tokens/s
TTFT
540
ms
LOCALSCORE
656
NVIDIA RTX 4000 Ada Generation
GPU / 20GB
PROMPT
2608
tokens/s
GENERATION
53.5
tokens/s
TTFT
520
ms
LOCALSCORE
645
NVIDIA RTX 3500 Ada Generation Laptop GPU
GPU / 12GB
PROMPT
2457
tokens/s
GENERATION
57.4
tokens/s
TTFT
540
ms
LOCALSCORE
639
NVIDIA GeForce RTX 2080 Ti
GPU / 11GB
PROMPT
2102
tokens/s
GENERATION
74.0
tokens/s
TTFT
601
ms
LOCALSCORE
637
NVIDIA TITAN V
GPU / 12GB
PROMPT
1823
tokens/s
GENERATION
73.3
tokens/s
TTFT
647
ms
LOCALSCORE
591
NVIDIA A100 80GB PCIe
GPU / 79GB
PROMPT
2601
tokens/s
GENERATION
37.1
tokens/s
TTFT
519
ms
LOCALSCORE
571
NVIDIA RTX A4000
GPU / 16GB
PROMPT
2000
tokens/s
GENERATION
60.3
tokens/s
TTFT
655
ms
LOCALSCORE
569
NVIDIA GeForce RTX 4060 Ti
GPU / 16GB
PROMPT
2215
tokens/s
GENERATION
48.2
tokens/s
TTFT
623
ms
LOCALSCORE
555
NVIDIA GeForce RTX 3060 Ti
GPU / 8GB
PROMPT
1885
tokens/s
GENERATION
61.4
tokens/s
TTFT
691
ms
LOCALSCORE
552
NVIDIA GeForce RTX 3070
GPU / 8GB
PROMPT
1955
tokens/s
GENERATION
57.1
tokens/s
TTFT
783
ms
LOCALSCORE
535
NVIDIA GeForce RTX 5070
GPU / 12GB
PROMPT
2360
tokens/s
GENERATION
37.0
tokens/s
TTFT
578
ms
LOCALSCORE
532
NVIDIA GeForce RTX 3080 Laptop GPU
GPU / 16GB
PROMPT
1885
tokens/s
GENERATION
52.3
tokens/s
TTFT
699
ms
LOCALSCORE
521
NVIDIA L4
GPU / 22GB
PROMPT
2056
tokens/s
GENERATION
42.9
tokens/s
TTFT
669
ms
LOCALSCORE
509
NVIDIA RTX 4000 SFF Ada Generation
GPU / 20GB
PROMPT
2013
tokens/s
GENERATION
44.6
tokens/s
TTFT
689
ms
LOCALSCORE
507
NVIDIA A10
GPU / 22GB
PROMPT
2085
tokens/s
GENERATION
36.5
tokens/s
TTFT
629
ms
LOCALSCORE
495
Quadro RTX 8000
GPU / 47GB
PROMPT
1677
tokens/s
GENERATION
74.0
tokens/s
TTFT
1.05
sec
LOCALSCORE
491
NVIDIA GeForce RTX 2080
GPU / 8GB
PROMPT
1593
tokens/s
GENERATION
57.1
tokens/s
TTFT
801
ms
LOCALSCORE
484
NVIDIA GeForce RTX 2070 SUPER
GPU / 8GB
PROMPT
1460
tokens/s
GENERATION
57.8
tokens/s
TTFT
880
ms
LOCALSCORE
458
NVIDIA GeForce RTX 3060
GPU / 12GB
PROMPT
1471
tokens/s
GENERATION
51.1
tokens/s
TTFT
894
ms
LOCALSCORE
438
NVIDIA GeForce RTX 4060 Ti
GPU / 8GB
PROMPT
2020
tokens/s
GENERATION
25.7
tokens/s
TTFT
641
ms
LOCALSCORE
431
NVIDIA RTX A5000 Laptop GPU
GPU / 16GB
PROMPT
1497
tokens/s
GENERATION
45.5
tokens/s
TTFT
917
ms
LOCALSCORE
420
Apple M3 Ultra 24P+8E+80GPU
GPU / 512GB
PROMPT
1100
tokens/s
GENERATION
62.6
tokens/s
TTFT
1.07
sec
LOCALSCORE
401
NVIDIA RTX PRO 2000 Blackwell Generation Laptop GPU
GPU / 8GB
PROMPT
1453
tokens/s
GENERATION
40.4
tokens/s
TTFT
920
ms
LOCALSCORE
400
Apple M3 Ultra 24P+8E+80GPU
GPU / 256GB
PROMPT
1062
tokens/s
GENERATION
63.3
tokens/s
TTFT
1.10
sec
LOCALSCORE
394
NVIDIA GeForce RTX 2060 SUPER
GPU / 8GB
PROMPT
1223
tokens/s
GENERATION
51.3
tokens/s
TTFT
1.04
sec
LOCALSCORE
392
NVIDIA GeForce RTX 4060
GPU / 8GB
PROMPT
1438
tokens/s
GENERATION
36.8
tokens/s
TTFT
1.25
sec
LOCALSCORE
375
NVIDIA RTX 2000 Ada Generation
GPU / 16GB
PROMPT
1357
tokens/s
GENERATION
37.5
tokens/s
TTFT
1.00
sec
LOCALSCORE
370
Quadro RTX 4000
GPU / 8GB
PROMPT
1096
tokens/s
GENERATION
41.2
tokens/s
TTFT
1.14
sec
LOCALSCORE
341
AMD Radeon RX 6900 XT
GPU / 16GB
PROMPT
1082
tokens/s
GENERATION
42.4
tokens/s
TTFT
1.31
sec
LOCALSCORE
328
AMD Radeon RX 7900 XTX
GPU / 24GB
PROMPT
888
tokens/s
GENERATION
51.2
tokens/s
TTFT
1.44
sec
LOCALSCORE
316
Radeon RX 7900 XTX
GPU / 24GB
PROMPT
885
tokens/s
GENERATION
45.3
tokens/s
TTFT
1.42
sec
LOCALSCORE
305
NVIDIA GeForce RTX 3060 Laptop GPU
GPU / 6GB
PROMPT
1110
tokens/s
GENERATION
37.3
tokens/s
TTFT
2.54
sec
LOCALSCORE
305
AMD Radeon RX 6800 XT
GPU / 16GB
PROMPT
926
tokens/s
GENERATION
35.5
tokens/s
TTFT
1.57
sec
LOCALSCORE
276
Apple M4 Max 12P+4E+40GPU
GPU / 48GB
PROMPT
663
tokens/s
GENERATION
55.1
tokens/s
TTFT
1.79
sec
LOCALSCORE
273
Apple M4 Max 12P+4E+40GPU
GPU / 128GB
PROMPT
622
tokens/s
GENERATION
51.4
tokens/s
TTFT
1.92
sec
LOCALSCORE
256
Apple M3 Max 12P+4E+40GPU
GPU / 128GB
PROMPT
591
tokens/s
GENERATION
45.6
tokens/s
TTFT
2.03
sec
LOCALSCORE
237
NVIDIA RTX A2000 8GB Laptop GPU
GPU / 8GB
PROMPT
790
tokens/s
GENERATION
27.5
tokens/s
TTFT
1.69
sec
LOCALSCORE
234
Apple M1 Ultra 16P+4E+48GPU
GPU / 128GB
PROMPT
534
tokens/s
GENERATION
48.9
tokens/s
TTFT
2.16
sec
LOCALSCORE
230
Apple M4 Max 10P+4E+32GPU
GPU / 36GB
PROMPT
551
tokens/s
GENERATION
47.2
tokens/s
TTFT
2.20
sec
LOCALSCORE
228
AMD Radeon RX 6800
GPU / 16GB
PROMPT
737
tokens/s
GENERATION
30.9
tokens/s
TTFT
1.95
sec
LOCALSCORE
227
NVIDIA RTX A2000
GPU / 6GB
PROMPT
758
tokens/s
GENERATION
28.7
tokens/s
TTFT
2.05
sec
LOCALSCORE
220
Apple M2 Max 8P+4E+38GPU
GPU / 96GB
PROMPT
484
tokens/s
GENERATION
46.4
tokens/s
TTFT
2.44
sec
LOCALSCORE
210
AMD Radeon RX 7800 XT
GPU / 16GB
PROMPT
524
tokens/s
GENERATION
38.9
tokens/s
TTFT
2.36
sec
LOCALSCORE
205
NVIDIA GeForce RTX 3050
GPU / 6GB
PROMPT
760
tokens/s
GENERATION
26.0
tokens/s
TTFT
2.32
sec
LOCALSCORE
204
Apple M2 Max 8P+4E+38GPU
GPU / 32GB
PROMPT
474
tokens/s
GENERATION
44.7
tokens/s
TTFT
2.49
sec
LOCALSCORE
204
Apple M3 Max 10P+4E+30GPU
GPU / 96GB
PROMPT
457
tokens/s
GENERATION
37.7
tokens/s
TTFT
2.69
sec
LOCALSCORE
186
Apple M3 Max 10P+4E+30GPU
GPU / 36GB
PROMPT
443
tokens/s
GENERATION
37.5
tokens/s
TTFT
2.73
sec
LOCALSCORE
183
AMD Radeon RX 6650 XT
GPU / 8GB
PROMPT
563
tokens/s
GENERATION
25.9
tokens/s
TTFT
2.48
sec
LOCALSCORE
181
AMD Radeon RX 6600 XT
GPU / 8GB
PROMPT
604
tokens/s
GENERATION
21.5
tokens/s
TTFT
2.34
sec
LOCALSCORE
177
Tesla P100-SXM2-16GB
GPU / 16GB
PROMPT
427
tokens/s
GENERATION
33.5
tokens/s
TTFT
2.72
sec
LOCALSCORE
174
Apple M1 Max 8P+2E+32GPU
GPU / 32GB
PROMPT
349
tokens/s
GENERATION
34.4
tokens/s
TTFT
3.58
sec
LOCALSCORE
150
Apple M4 Pro 10P+4E+20GPU
GPU / 48GB
PROMPT
357
tokens/s
GENERATION
32.5
tokens/s
TTFT
3.47
sec
LOCALSCORE
150
AMD Radeon RX 6700 XT
GPU / 12GB
PROMPT
597
tokens/s
GENERATION
16.4
tokens/s
TTFT
2.43
sec
LOCALSCORE
149
Apple M4 Pro 10P+4E+20GPU
GPU / 24GB
PROMPT
351
tokens/s
GENERATION
31.6
tokens/s
TTFT
3.45
sec
LOCALSCORE
148
Apple M3 Max 12P+4E+40GPU
GPU / 64GB
PROMPT
377
tokens/s
GENERATION
25.4
tokens/s
TTFT
3.37
sec
LOCALSCORE
142
Apple M4 Pro 10P+4E+20GPU
GPU / 64GB
PROMPT
329
tokens/s
GENERATION
31.8
tokens/s
TTFT
3.79
sec
LOCALSCORE
140
Apple M1 Max 8P+2E+24GPU
GPU / 64GB
PROMPT
306
tokens/s
GENERATION
32.1
tokens/s
TTFT
3.99
sec
LOCALSCORE
135
Apple M4 Pro 8P+4E+16GPU
GPU / 48GB
PROMPT
302
tokens/s
GENERATION
30.2
tokens/s
TTFT
4.14
sec
LOCALSCORE
130
NVIDIA GeForce RTX 4060 Laptop GPU
GPU / 8GB
PROMPT
809
tokens/s
GENERATION
24.3
tokens/s
TTFT
9.09
sec
LOCALSCORE
129
Apple M4 Pro 8P+4E+16GPU
GPU / 24GB
PROMPT
292
tokens/s
GENERATION
30.5
tokens/s
TTFT
4.30
sec
LOCALSCORE
127
Apple M2 Pro 8P+4E+19GPU
GPU / 32GB
PROMPT
262
tokens/s
GENERATION
26.3
tokens/s
TTFT
4.70
sec
LOCALSCORE
114
Apple M3 Pro 6P+6E+18GPU
GPU / 36GB
PROMPT
284
tokens/s
GENERATION
22.1
tokens/s
TTFT
4.46
sec
LOCALSCORE
112
AMD Radeon RX 5700 XT
GPU / 8GB
PROMPT
256
tokens/s
GENERATION
25.4
tokens/s
TTFT
5.68
sec
LOCALSCORE
105
Apple M2 Pro 6P+4E+16GPU
GPU / 16GB
PROMPT
229
tokens/s
GENERATION
25.0
tokens/s
TTFT
5.43
sec
LOCALSCORE
102
Apple M3 Pro 5P+6E+14GPU
GPU / 36GB
PROMPT
223
tokens/s
GENERATION
21.5
tokens/s
TTFT
5.65
sec
LOCALSCORE
95
Apple M2 Pro 6P+4E+16GPU
GPU / 32GB
PROMPT
209
tokens/s
GENERATION
23.8
tokens/s
TTFT
6.07
sec
LOCALSCORE
94
Apple M1 Pro 8P+2E+16GPU
GPU / 16GB
PROMPT
204
tokens/s
GENERATION
21.5
tokens/s
TTFT
6.11
sec
LOCALSCORE
89
Apple M1 Pro 8P+2E+16GPU
GPU / 32GB
PROMPT
200
tokens/s
GENERATION
21.6
tokens/s
TTFT
6.25
sec
LOCALSCORE
88
Apple M3 Pro 5P+6E+14GPU
GPU / 18GB
PROMPT
200
tokens/s
GENERATION
19.1
tokens/s
TTFT
6.43
sec
LOCALSCORE
84
Apple M1 Pro 6P+2E+14GPU
GPU / 16GB
PROMPT
176
tokens/s
GENERATION
20.1
tokens/s
TTFT
7.05
sec
LOCALSCORE
79
Apple M4 4P+6E+10GPU
GPU / 16GB
PROMPT
186
tokens/s
GENERATION
17.7
tokens/s
TTFT
6.87
sec
LOCALSCORE
78
Apple M1 Pro 6P+2E+14GPU
GPU / 32GB
PROMPT
173
tokens/s
GENERATION
20.0
tokens/s
TTFT
7.23
sec
LOCALSCORE
78
Apple M2 4P+4E+8GPU
GPU / 8GB
PROMPT
148
tokens/s
GENERATION
18.8
tokens/s
TTFT
8.40
sec
LOCALSCORE
69
Apple M4 4P+6E+10GPU
GPU / 24GB
PROMPT
159
tokens/s
GENERATION
16.6
tokens/s
TTFT
7.93
sec
LOCALSCORE
69
Apple M4 4P+6E+10GPU
GPU / 32GB
PROMPT
146
tokens/s
GENERATION
15.9
tokens/s
TTFT
8.26
sec
LOCALSCORE
65
Apple M2 4P+4E+10GPU
GPU / 16GB
PROMPT
143
tokens/s
GENERATION
14.9
tokens/s
TTFT
8.96
sec
LOCALSCORE
62
Apple M2 4P+4E+10GPU
GPU / 24GB
PROMPT
142
tokens/s
GENERATION
14.7
tokens/s
TTFT
9.02
sec
LOCALSCORE
61
Apple M3 4P+4E+10GPU
GPU / 16GB
PROMPT
137
tokens/s
GENERATION
13.6
tokens/s
TTFT
9.00
sec
LOCALSCORE
59
NVIDIA GeForce RTX 3050 Ti Laptop GPU
GPU / 4GB
PROMPT
290
tokens/s
GENERATION
4.3
tokens/s
TTFT
8.73
sec
LOCALSCORE
52
NVIDIA GeForce GTX 1660
GPU / 6GB
PROMPT
129
tokens/s
GENERATION
9.0
tokens/s
TTFT
11.11
sec
LOCALSCORE
47
Apple M3 4P+4E+10GPU
GPU / 24GB
PROMPT
110
tokens/s
GENERATION
10.2
tokens/s
TTFT
11.67
sec
LOCALSCORE
46
Apple M1 4P+4E+7GPU
GPU / 16GB
PROMPT
80
tokens/s
GENERATION
9.1
tokens/s
TTFT
15.23
sec
LOCALSCORE
36
NVIDIA GeForce GTX 1650 Ti
GPU / 4GB
PROMPT
66
tokens/s
GENERATION
2.8
tokens/s
TTFT
23.14
sec
LOCALSCORE
20