logo
HomeLatest ResultsDownloadAboutBlog
model icon

Qwen2.5 14B Instruct

Q4_K - Medium

14.8Bparams
COMPARE ACCELERATORS

371 accelerators tested

Select Accelerators

Apple M3 4P+4E+10GPU

16GB

NVIDIA RTX PRO 6000 Blackwell Workstation Edition

95GB

NVIDIA GeForce RTX 4090

23GB

NVIDIA GeForce RTX 5090

32GB

NVIDIA GeForce RTX 4090 D

47GB

Qwen2.5 14B Instruct - Q4_K - Medium

LEADERBOARD
Apple M3 4P+4E+10GPU
GPU / 16GB
PROMPT
4916
tokens/s
GENERATION
784
tokens/s
TTFT
14.45
sec
LOCALSCORE
2523
NVIDIA RTX PRO 6000 Blackwell Workstation Edition
GPU / 95GB
PROMPT
4761
tokens/s
GENERATION
70.8
tokens/s
TTFT
281
ms
LOCALSCORE
1080
NVIDIA GeForce RTX 4090
GPU / 23GB
PROMPT
4208
tokens/s
GENERATION
68.8
tokens/s
TTFT
315
ms
LOCALSCORE
972
NVIDIA GeForce RTX 5090
GPU / 32GB
PROMPT
4898
tokens/s
GENERATION
55.8
tokens/s
TTFT
374
ms
LOCALSCORE
950
NVIDIA GeForce RTX 4090 D
GPU / 47GB
PROMPT
3905
tokens/s
GENERATION
70.5
tokens/s
TTFT
336
ms
LOCALSCORE
935
NVIDIA RTX 6000 Ada Generation
GPU / 47GB
PROMPT
3552
tokens/s
GENERATION
70.0
tokens/s
TTFT
369
ms
LOCALSCORE
877
NVIDIA GeForce RTX 4090
GPU / 22GB
PROMPT
4028
tokens/s
GENERATION
51.8
tokens/s
TTFT
334
ms
LOCALSCORE
854
NVIDIA H100 PCIe
GPU / 79GB
PROMPT
2973
tokens/s
GENERATION
72.2
tokens/s
TTFT
417
ms
LOCALSCORE
802
NVIDIA GeForce RTX 4090
GPU / 24GB
PROMPT
3506
tokens/s
GENERATION
49.1
tokens/s
TTFT
396
ms
LOCALSCORE
766
NVIDIA GeForce RTX 5090
GPU / 31GB
PROMPT
3678
tokens/s
GENERATION
45.5
tokens/s
TTFT
536
ms
LOCALSCORE
708
NVIDIA GeForce RTX 4080 SUPER
GPU / 32GB
PROMPT
2803
tokens/s
GENERATION
46.2
tokens/s
TTFT
468
ms
LOCALSCORE
651
NVIDIA GeForce RTX 3090 Ti
GPU / 24GB
PROMPT
2202
tokens/s
GENERATION
64.2
tokens/s
TTFT
575
ms
LOCALSCORE
627
NVIDIA A100-SXM4-80GB
GPU / 79GB
PROMPT
2069
tokens/s
GENERATION
66.3
tokens/s
TTFT
592
ms
LOCALSCORE
614
NVIDIA GeForce RTX 4080 SUPER
GPU / 16GB
PROMPT
2681
tokens/s
GENERATION
43.6
tokens/s
TTFT
503
ms
LOCALSCORE
614
NVIDIA RTX 6000 Ada Generation
GPU / 48GB
PROMPT
2936
tokens/s
GENERATION
28.1
tokens/s
TTFT
471
ms
LOCALSCORE
560
NVIDIA GeForce RTX 3090
GPU / 24GB
PROMPT
1942
tokens/s
GENERATION
55.7
tokens/s
TTFT
662
ms
LOCALSCORE
547
NVIDIA GeForce RTX 4070 Ti
GPU / 12GB
PROMPT
2203
tokens/s
GENERATION
40.0
tokens/s
TTFT
616
ms
LOCALSCORE
523
NVIDIA RTX A6000
GPU / 48GB
PROMPT
1883
tokens/s
GENERATION
52.0
tokens/s
TTFT
692
ms
LOCALSCORE
521
NVIDIA GeForce RTX 4080
GPU / 16GB
PROMPT
2339
tokens/s
GENERATION
35.3
tokens/s
TTFT
594
ms
LOCALSCORE
519
NVIDIA L40
GPU / 44GB
PROMPT
2713
tokens/s
GENERATION
24.7
tokens/s
TTFT
513
ms
LOCALSCORE
508
NVIDIA GeForce RTX 4070 Ti SUPER
GPU / 16GB
PROMPT
2147
tokens/s
GENERATION
36.8
tokens/s
TTFT
617
ms
LOCALSCORE
505
NVIDIA H100 80GB HBM3
GPU / 79GB
PROMPT
2609
tokens/s
GENERATION
23.4
tokens/s
TTFT
475
ms
LOCALSCORE
505
NVIDIA GeForce RTX 5070 Ti
GPU / 15GB
PROMPT
2193
tokens/s
GENERATION
37.0
tokens/s
TTFT
674
ms
LOCALSCORE
496
NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
GPU / 95GB
PROMPT
2623
tokens/s
GENERATION
23.7
tokens/s
TTFT
544
ms
LOCALSCORE
494
NVIDIA A40
GPU / 44GB
PROMPT
1825
tokens/s
GENERATION
46.2
tokens/s
TTFT
718
ms
LOCALSCORE
490
NVIDIA GeForce RTX 3080 Ti
GPU / 12GB
PROMPT
1793
tokens/s
GENERATION
46.1
tokens/s
TTFT
732
ms
LOCALSCORE
483
NVIDIA GeForce RTX 3080
GPU / 20GB
PROMPT
1741
tokens/s
GENERATION
46.6
tokens/s
TTFT
734
ms
LOCALSCORE
480
NVIDIA L40S
GPU / 45GB
PROMPT
2868
tokens/s
GENERATION
19.3
tokens/s
TTFT
495
ms
LOCALSCORE
477
NVIDIA RTX A5000
GPU / 24GB
PROMPT
1532
tokens/s
GENERATION
50.4
tokens/s
TTFT
837
ms
LOCALSCORE
452
NVIDIA H100 NVL
GPU / 93GB
PROMPT
2354
tokens/s
GENERATION
20.5
tokens/s
TTFT
528
ms
LOCALSCORE
450
NVIDIA A40-48Q
GPU / 48GB
PROMPT
1707
tokens/s
GENERATION
39.5
tokens/s
TTFT
762
ms
LOCALSCORE
446
NVIDIA RTX PRO 6000 Blackwell Server Edition
GPU / 95GB
PROMPT
2581
tokens/s
GENERATION
17.9
tokens/s
TTFT
637
ms
LOCALSCORE
438
NVIDIA GeForce RTX 4070
GPU / 12GB
PROMPT
1692
tokens/s
GENERATION
37.9
tokens/s
TTFT
779
ms
LOCALSCORE
435
NVIDIA GeForce RTX 5080
GPU / 15GB
PROMPT
2097
tokens/s
GENERATION
25.5
tokens/s
TTFT
768
ms
LOCALSCORE
422
NVIDIA GeForce RTX 5090 Laptop GPU
GPU / 24GB
PROMPT
1550
tokens/s
GENERATION
40.5
tokens/s
TTFT
866
ms
LOCALSCORE
417
NVIDIA GeForce RTX 5070
GPU / 11GB
PROMPT
1758
tokens/s
GENERATION
45.7
tokens/s
TTFT
1.19
sec
LOCALSCORE
407
Tesla V100-SXM2-16GB
GPU / 16GB
PROMPT
1184
tokens/s
GENERATION
53.0
tokens/s
TTFT
991
ms
LOCALSCORE
399
NVIDIA RTX A4500
GPU / 20GB
PROMPT
1331
tokens/s
GENERATION
44.0
tokens/s
TTFT
977
ms
LOCALSCORE
391
Tesla V100-SXM2-32GB
GPU / 32GB
PROMPT
1178
tokens/s
GENERATION
49.1
tokens/s
TTFT
996
ms
LOCALSCORE
387
NVIDIA GeForce RTX 5070 Ti Laptop GPU
GPU / 12GB
PROMPT
1403
tokens/s
GENERATION
36.8
tokens/s
TTFT
925
ms
LOCALSCORE
382
NVIDIA GeForce RTX 5070 Ti
GPU / 16GB
PROMPT
1727
tokens/s
GENERATION
24.5
tokens/s
TTFT
816
ms
LOCALSCORE
382
NVIDIA GeForce RTX 4070 SUPER
GPU / 12GB
PROMPT
1701
tokens/s
GENERATION
24.9
tokens/s
TTFT
786
ms
LOCALSCORE
377
NVIDIA RTX PRO 4500 Blackwell
GPU / 31GB
PROMPT
1824
tokens/s
GENERATION
24.2
tokens/s
TTFT
858
ms
LOCALSCORE
372
NVIDIA TITAN RTX
GPU / 23GB
PROMPT
1200
tokens/s
GENERATION
42.8
tokens/s
TTFT
1.05
sec
LOCALSCORE
365
Quadro RTX 8000
GPU / 47GB
PROMPT
1178
tokens/s
GENERATION
42.9
tokens/s
TTFT
1.06
sec
LOCALSCORE
363
NVIDIA RTX 4000 Ada Generation
GPU / 20GB
PROMPT
1417
tokens/s
GENERATION
30.0
tokens/s
TTFT
944
ms
LOCALSCORE
356
NVIDIA GeForce RTX 2080 Ti
GPU / 11GB
PROMPT
1140
tokens/s
GENERATION
39.5
tokens/s
TTFT
1.09
sec
LOCALSCORE
346
NVIDIA GeForce RTX 5060 Ti
GPU / 15GB
PROMPT
1335
tokens/s
GENERATION
32.9
tokens/s
TTFT
1.25
sec
LOCALSCORE
338
Quadro RTX 8000
GPU / 44GB
PROMPT
1087
tokens/s
GENERATION
36.7
tokens/s
TTFT
1.17
sec
LOCALSCORE
324
Quadro RTX 6000
GPU / 22GB
PROMPT
1077
tokens/s
GENERATION
37.1
tokens/s
TTFT
1.18
sec
LOCALSCORE
323
NVIDIA TITAN V
GPU / 12GB
PROMPT
962
tokens/s
GENERATION
42.2
tokens/s
TTFT
1.21
sec
LOCALSCORE
323
NVIDIA A100 80GB PCIe
GPU / 79GB
PROMPT
1438
tokens/s
GENERATION
20.8
tokens/s
TTFT
954
ms
LOCALSCORE
315
NVIDIA RTX 3500 Ada Generation Laptop GPU
GPU / 12GB
PROMPT
1298
tokens/s
GENERATION
25.1
tokens/s
TTFT
1.02
sec
LOCALSCORE
315
NVIDIA GeForce RTX 4060 Ti
GPU / 16GB
PROMPT
1239
tokens/s
GENERATION
25.6
tokens/s
TTFT
1.10
sec
LOCALSCORE
306
NVIDIA GeForce RTX 5060 Ti
GPU / 16GB
PROMPT
1284
tokens/s
GENERATION
22.6
tokens/s
TTFT
1.03
sec
LOCALSCORE
304
NVIDIA RTX A4000
GPU / 16GB
PROMPT
1042
tokens/s
GENERATION
32.0
tokens/s
TTFT
1.24
sec
LOCALSCORE
299
NVIDIA GeForce RTX 5070
GPU / 12GB
PROMPT
1264
tokens/s
GENERATION
20.8
tokens/s
TTFT
1.07
sec
LOCALSCORE
290
Tesla V100-PCIE-32GB
GPU / 32GB
PROMPT
907
tokens/s
GENERATION
28.9
tokens/s
TTFT
1.21
sec
LOCALSCORE
279
NVIDIA L4
GPU / 22GB
PROMPT
1095
tokens/s
GENERATION
23.6
tokens/s
TTFT
1.24
sec
LOCALSCORE
275
NVIDIA RTX 4000 SFF Ada Generation
GPU / 20GB
PROMPT
1051
tokens/s
GENERATION
24.7
tokens/s
TTFT
1.30
sec
LOCALSCORE
271
NVIDIA A10
GPU / 22GB
PROMPT
1133
tokens/s
GENERATION
19.9
tokens/s
TTFT
1.16
sec
LOCALSCORE
269
NVIDIA GB10
GPU / 120GB
PROMPT
1119
tokens/s
GENERATION
18.4
tokens/s
TTFT
1.21
sec
LOCALSCORE
257
NVIDIA GeForce RTX 3080 Laptop GPU
GPU / 16GB
PROMPT
858
tokens/s
GENERATION
28.2
tokens/s
TTFT
1.46
sec
LOCALSCORE
255
NVIDIA GB10
GPU / 122GB
PROMPT
1094
tokens/s
GENERATION
18.3
tokens/s
TTFT
1.26
sec
LOCALSCORE
252
NVIDIA RTX A5000 Laptop GPU
GPU / 16GB
PROMPT
839
tokens/s
GENERATION
27.6
tokens/s
TTFT
1.55
sec
LOCALSCORE
246
NVIDIA GeForce RTX 3080
GPU / 10GB
PROMPT
1040
tokens/s
GENERATION
33.9
tokens/s
TTFT
5.84
sec
LOCALSCORE
224
NVIDIA GeForce RTX 3060
GPU / 12GB
PROMPT
753
tokens/s
GENERATION
26.4
tokens/s
TTFT
1.96
sec
LOCALSCORE
223
NVIDIA RTX 5000 Ada Generation Laptop GPU
GPU / 16GB
PROMPT
952
tokens/s
GENERATION
16.4
tokens/s
TTFT
1.44
sec
LOCALSCORE
221
Apple M3 Ultra 24P+8E+80GPU
GPU / 512GB
PROMPT
580
tokens/s
GENERATION
35.8
tokens/s
TTFT
2.04
sec
LOCALSCORE
217
Apple M3 Ultra 24P+8E+80GPU
GPU / 256GB
PROMPT
568
tokens/s
GENERATION
36.7
tokens/s
TTFT
2.08
sec
LOCALSCORE
216
NVIDIA RTX 2000 Ada Generation
GPU / 16GB
PROMPT
740
tokens/s
GENERATION
20.7
tokens/s
TTFT
1.81
sec
LOCALSCORE
204
Apple M2 Ultra 16P+8E+76GPU
GPU / 128GB
PROMPT
471
tokens/s
GENERATION
36.6
tokens/s
TTFT
2.48
sec
LOCALSCORE
191
Apple M2 Ultra 16P+8E+76GPU
GPU / 192GB
PROMPT
469
tokens/s
GENERATION
36.5
tokens/s
TTFT
2.49
sec
LOCALSCORE
190
NVIDIA GeForce RTX 2060
GPU / 12GB
PROMPT
595
tokens/s
GENERATION
22.8
tokens/s
TTFT
2.16
sec
LOCALSCORE
184
Apple M3 Ultra 20P+8E+60GPU
GPU / 96GB
PROMPT
445
tokens/s
GENERATION
34.4
tokens/s
TTFT
2.67
sec
LOCALSCORE
179
AMD Radeon RX 7900 XTX
GPU / 24GB
PROMPT
488
tokens/s
GENERATION
30.9
tokens/s
TTFT
2.64
sec
LOCALSCORE
179
Radeon RX 7900 XTX
GPU / 24GB
PROMPT
502
tokens/s
GENERATION
27.9
tokens/s
TTFT
2.57
sec
LOCALSCORE
176
Apple M2 Ultra 16P+8E+60GPU
GPU / 64GB
PROMPT
381
tokens/s
GENERATION
34.2
tokens/s
TTFT
3.06
sec
LOCALSCORE
162
Apple M2 Ultra 16P+8E+60GPU
GPU / 128GB
PROMPT
377
tokens/s
GENERATION
33.6
tokens/s
TTFT
3.10
sec
LOCALSCORE
160
NVIDIA RTX A2000 12GB
GPU / 12GB
PROMPT
506
tokens/s
GENERATION
19.5
tokens/s
TTFT
2.52
sec
LOCALSCORE
158
Apple M1 Ultra 16P+4E+64GPU
GPU / 128GB
PROMPT
372
tokens/s
GENERATION
32.4
tokens/s
TTFT
3.13
sec
LOCALSCORE
157
Apple M5 Max 6P+12E+40GPU
GPU / 128GB
PROMPT
353
tokens/s
GENERATION
36.7
tokens/s
TTFT
3.39
sec
LOCALSCORE
156
AMD Radeon RX 6800 XT
GPU / 16GB
PROMPT
505
tokens/s
GENERATION
20.9
tokens/s
TTFT
2.86
sec
LOCALSCORE
155
AMD Radeon RX 6900 XT
GPU / 16GB
PROMPT
569
tokens/s
GENERATION
18.5
tokens/s
TTFT
2.63
sec
LOCALSCORE
153
Apple M5 Max 6P+12E+32GPU
GPU / 36GB
PROMPT
342
tokens/s
GENERATION
34.0
tokens/s
TTFT
3.58
sec
LOCALSCORE
148
Apple M4 Max 12P+4E+40GPU
GPU / 48GB
PROMPT
340
tokens/s
GENERATION
30.2
tokens/s
TTFT
3.58
sec
LOCALSCORE
142
NVIDIA TITAN Xp
GPU / 12GB
PROMPT
412
tokens/s
GENERATION
21.1
tokens/s
TTFT
3.09
sec
LOCALSCORE
141
NVIDIA GeForce GTX 1080 Ti
GPU / 11GB
PROMPT
419
tokens/s
GENERATION
20.2
tokens/s
TTFT
3.05
sec
LOCALSCORE
141
Apple M4 Max 12P+4E+40GPU
GPU / 128GB
PROMPT
327
tokens/s
GENERATION
28.7
tokens/s
TTFT
3.79
sec
LOCALSCORE
136
Tesla T4
GPU / 15GB
PROMPT
450
tokens/s
GENERATION
13.3
tokens/s
TTFT
2.76
sec
LOCALSCORE
130
Apple M1 Ultra 16P+4E+48GPU
GPU / 128GB
PROMPT
290
tokens/s
GENERATION
27.8
tokens/s
TTFT
4.04
sec
LOCALSCORE
126
Apple M3 Max 12P+4E+40GPU
GPU / 128GB
PROMPT
302
tokens/s
GENERATION
25.5
tokens/s
TTFT
4.07
sec
LOCALSCORE
124
Apple M4 Max 12P+4E+40GPU
GPU / 64GB
PROMPT
289
tokens/s
GENERATION
26.3
tokens/s
TTFT
4.44
sec
LOCALSCORE
121
AMD Radeon RX 7800 XT
GPU / 16GB
PROMPT
302
tokens/s
GENERATION
23.0
tokens/s
TTFT
4.16
sec
LOCALSCORE
119
Apple M4 Max 10P+4E+32GPU
GPU / 36GB
PROMPT
282
tokens/s
GENERATION
25.3
tokens/s
TTFT
4.36
sec
LOCALSCORE
118
Apple M3 Max 12P+4E+40GPU
GPU / 48GB
PROMPT
287
tokens/s
GENERATION
23.6
tokens/s
TTFT
4.27
sec
LOCALSCORE
117
AMD Radeon RX 6800
GPU / 16GB
PROMPT
399
tokens/s
GENERATION
14.8
tokens/s
TTFT
3.68
sec
LOCALSCORE
112
Tesla P40
GPU / 24GB
PROMPT
329
tokens/s
GENERATION
15.4
tokens/s
TTFT
3.74
sec
LOCALSCORE
112
Apple M2 Max 8P+4E+38GPU
GPU / 96GB
PROMPT
252
tokens/s
GENERATION
25.2
tokens/s
TTFT
4.80
sec
LOCALSCORE
110
AMD Radeon RX 7800 XT
GPU / 16GB
PROMPT
272
tokens/s
GENERATION
22.5
tokens/s
TTFT
5.45
sec
LOCALSCORE
107
Quadro P5000
GPU / 16GB
PROMPT
303
tokens/s
GENERATION
14.9
tokens/s
TTFT
4.28
sec
LOCALSCORE
102
AMD Radeon RX 6700 XT
GPU / 12GB
PROMPT
316
tokens/s
GENERATION
15.7
tokens/s
TTFT
4.94
sec
LOCALSCORE
100
Apple M3 Max 10P+4E+30GPU
GPU / 96GB
PROMPT
239
tokens/s
GENERATION
20.8
tokens/s
TTFT
5.16
sec
LOCALSCORE
99
Apple M2 Max 8P+4E+38GPU
GPU / 32GB
PROMPT
226
tokens/s
GENERATION
20.6
tokens/s
TTFT
4.98
sec
LOCALSCORE
98
Apple M5 Pro 6P+12E+20GPU
GPU / 24GB
PROMPT
217
tokens/s
GENERATION
24.2
tokens/s
TTFT
5.76
sec
LOCALSCORE
97
Apple M5 Pro 6P+12E+20GPU
GPU / 64GB
PROMPT
218
tokens/s
GENERATION
24.0
tokens/s
TTFT
5.77
sec
LOCALSCORE
97
Apple M2 Max 8P+4E+38GPU
GPU / 64GB
PROMPT
223
tokens/s
GENERATION
22.0
tokens/s
TTFT
5.51
sec
LOCALSCORE
96
Apple M5 Pro 6P+12E+20GPU
GPU / 48GB
PROMPT
216
tokens/s
GENERATION
23.7
tokens/s
TTFT
5.80
sec
LOCALSCORE
96
Apple M3 Max 10P+4E+30GPU
GPU / 36GB
PROMPT
226
tokens/s
GENERATION
19.8
tokens/s
TTFT
5.41
sec
LOCALSCORE
94
Tesla P100-SXM2-16GB
GPU / 16GB
PROMPT
204
tokens/s
GENERATION
20.5
tokens/s
TTFT
5.71
sec
LOCALSCORE
90
Apple M3 Max 12P+4E+40GPU
GPU / 64GB
PROMPT
233
tokens/s
GENERATION
17.2
tokens/s
TTFT
5.80
sec
LOCALSCORE
90
AMD Radeon RX 6950 XT
GPU / 16GB
PROMPT
549
tokens/s
GENERATION
3.3
tokens/s
TTFT
2.85
sec
LOCALSCORE
86
Apple M1 Max 8P+2E+32GPU
GPU / 32GB
PROMPT
197
tokens/s
GENERATION
20.2
tokens/s
TTFT
6.26
sec
LOCALSCORE
86
NVIDIA GeForce RTX 3060 Ti
GPU / 8GB
PROMPT
430
tokens/s
GENERATION
7.5
tokens/s
TTFT
5.55
sec
LOCALSCORE
83
Apple M1 Max 8P+2E+32GPU
GPU / 64GB
PROMPT
189
tokens/s
GENERATION
19.1
tokens/s
TTFT
6.74
sec
LOCALSCORE
82
Tesla P100-PCIE-16GB
GPU / 16GB
PROMPT
181
tokens/s
GENERATION
19.2
tokens/s
TTFT
6.43
sec
LOCALSCORE
81
Apple M4 Pro 10P+4E+20GPU
GPU / 24GB
PROMPT
190
tokens/s
GENERATION
18.0
tokens/s
TTFT
6.63
sec
LOCALSCORE
80
Apple M4 Pro 10P+4E+20GPU
GPU / 48GB
PROMPT
190
tokens/s
GENERATION
18.0
tokens/s
TTFT
6.62
sec
LOCALSCORE
80
NVIDIA RTX A4500 Laptop GPU
GPU / 15GB
PROMPT
302
tokens/s
GENERATION
7.9
tokens/s
TTFT
4.77
sec
LOCALSCORE
80
Apple M5 Pro 5P+10E+16GPU
GPU / 24GB
PROMPT
169
tokens/s
GENERATION
21.6
tokens/s
TTFT
7.56
sec
LOCALSCORE
79
Apple M4 Pro 10P+4E+20GPU
GPU / 64GB
PROMPT
183
tokens/s
GENERATION
18.0
tokens/s
TTFT
6.92
sec
LOCALSCORE
78
Apple M1 Max 8P+2E+24GPU
GPU / 32GB
PROMPT
157
tokens/s
GENERATION
18.8
tokens/s
TTFT
7.80
sec
LOCALSCORE
72
NVIDIA GeForce RTX 3070
GPU / 8GB
PROMPT
398
tokens/s
GENERATION
5.6
tokens/s
TTFT
6.59
sec
LOCALSCORE
71
Apple M4 Pro 8P+4E+16GPU
GPU / 48GB
PROMPT
157
tokens/s
GENERATION
16.7
tokens/s
TTFT
8.06
sec
LOCALSCORE
69
Apple M4 Pro 8P+4E+16GPU
GPU / 64GB
PROMPT
151
tokens/s
GENERATION
16.1
tokens/s
TTFT
8.46
sec
LOCALSCORE
66
Apple M2 Max 8P+4E+30GPU
GPU / 64GB
PROMPT
149
tokens/s
GENERATION
14.5
tokens/s
TTFT
7.68
sec
LOCALSCORE
66
Apple M4 Pro 8P+4E+16GPU
GPU / 24GB
PROMPT
149
tokens/s
GENERATION
15.9
tokens/s
TTFT
8.56
sec
LOCALSCORE
65
Apple M1 Max 8P+2E+24GPU
GPU / 64GB
PROMPT
140
tokens/s
GENERATION
15.1
tokens/s
TTFT
8.83
sec
LOCALSCORE
63
NVIDIA GeForce RTX 2080
GPU / 8GB
PROMPT
324
tokens/s
GENERATION
5.3
tokens/s
TTFT
7.87
sec
LOCALSCORE
60
Apple M2 Pro 8P+4E+19GPU
GPU / 32GB
PROMPT
137
tokens/s
GENERATION
14.1
tokens/s
TTFT
9.10
sec
LOCALSCORE
60
NVIDIA L40S-8Q
GPU / 8GB
PROMPT
343
tokens/s
GENERATION
3.5
tokens/s
TTFT
5.74
sec
LOCALSCORE
59
Apple M3 Pro 6P+6E+18GPU
GPU / 36GB
PROMPT
147
tokens/s
GENERATION
11.9
tokens/s
TTFT
8.75
sec
LOCALSCORE
59
Apple M3 Pro 6P+6E+18GPU
GPU / 18GB
PROMPT
145
tokens/s
GENERATION
11.8
tokens/s
TTFT
8.83
sec
LOCALSCORE
58
NVIDIA GeForce RTX 2080 SUPER
GPU / 8GB
PROMPT
294
tokens/s
GENERATION
4.5
tokens/s
TTFT
8.27
sec
LOCALSCORE
54
Apple M2 Pro 6P+4E+16GPU
GPU / 16GB
PROMPT
118
tokens/s
GENERATION
13.3
tokens/s
TTFT
10.66
sec
LOCALSCORE
53
NVIDIA GeForce RTX 4060 Ti
GPU / 8GB
PROMPT
327
tokens/s
GENERATION
3.5
tokens/s
TTFT
7.99
sec
LOCALSCORE
52
Apple M3 Pro 5P+6E+14GPU
GPU / 36GB
PROMPT
120
tokens/s
GENERATION
12.1
tokens/s
TTFT
10.65
sec
LOCALSCORE
51
NVIDIA GeForce RTX 4070 Laptop GPU
GPU / 8GB
PROMPT
289
tokens/s
GENERATION
3.8
tokens/s
TTFT
8.84
sec
LOCALSCORE
50
Apple M3 Pro 5P+6E+14GPU
GPU / 18GB
PROMPT
115
tokens/s
GENERATION
11.6
tokens/s
TTFT
11.25
sec
LOCALSCORE
49
Apple M5 4P+6E+10GPU
GPU / 32GB
PROMPT
110
tokens/s
GENERATION
11.5
tokens/s
TTFT
11.64
sec
LOCALSCORE
48
Apple M1 Pro 8P+2E+16GPU
GPU / 16GB
PROMPT
107
tokens/s
GENERATION
11.9
tokens/s
TTFT
11.84
sec
LOCALSCORE
48
Apple M1 Pro 8P+2E+16GPU
GPU / 32GB
PROMPT
104
tokens/s
GENERATION
11.6
tokens/s
TTFT
12.08
sec
LOCALSCORE
46
NVIDIA A2
GPU / 15GB
PROMPT
160
tokens/s
GENERATION
5.6
tokens/s
TTFT
10.52
sec
LOCALSCORE
44
Apple M1 Pro 6P+2E+14GPU
GPU / 16GB
PROMPT
92
tokens/s
GENERATION
10.8
tokens/s
TTFT
13.49
sec
LOCALSCORE
42
Apple M1 Pro 8P+2E+14GPU
GPU / 32GB
PROMPT
89
tokens/s
GENERATION
10.4
tokens/s
TTFT
14.17
sec
LOCALSCORE
40
Apple M4 4P+6E+10GPU
GPU / 24GB
PROMPT
92
tokens/s
GENERATION
9.1
tokens/s
TTFT
14.01
sec
LOCALSCORE
39
Apple M4 4P+6E+10GPU
GPU / 16GB
PROMPT
86
tokens/s
GENERATION
9.0
tokens/s
TTFT
15.12
sec
LOCALSCORE
37
Apple M1 Pro 6P+2E+14GPU
GPU / 32GB
PROMPT
83
tokens/s
GENERATION
8.9
tokens/s
TTFT
15.58
sec
LOCALSCORE
36
Apple M4 4P+6E+10GPU
GPU / 32GB
PROMPT
79
tokens/s
GENERATION
8.6
tokens/s
TTFT
15.82
sec
LOCALSCORE
35
NVIDIA RTX PRO 2000 Blackwell Generation Laptop GPU
GPU / 8GB
PROMPT
230
tokens/s
GENERATION
2.1
tokens/s
TTFT
14.47
sec
LOCALSCORE
32
Apple M2 4P+4E+10GPU
GPU / 16GB
PROMPT
72
tokens/s
GENERATION
7.7
tokens/s
TTFT
17.67
sec
LOCALSCORE
32
Apple M2 4P+4E+10GPU
GPU / 24GB
PROMPT
70
tokens/s
GENERATION
7.3
tokens/s
TTFT
18.03
sec
LOCALSCORE
31
AMD Radeon Graphics
GPU / 32GB
PROMPT
51
tokens/s
GENERATION
14.1
tokens/s
TTFT
28.62
sec
LOCALSCORE
29
AMD Radeon 780M Graphics
GPU / 28GB
PROMPT
99
tokens/s
GENERATION
3.1
tokens/s
TTFT
13.29
sec
LOCALSCORE
28
Apple M4 4P+6E+8GPU
GPU / 16GB
PROMPT
61
tokens/s
GENERATION
7.3
tokens/s
TTFT
20.17
sec
LOCALSCORE
28
AMD Instinct MI50/MI60
GPU / 16GB
PROMPT
48
tokens/s
GENERATION
12.9
tokens/s
TTFT
30.10
sec
LOCALSCORE
27
Apple M3 4P+4E+10GPU
GPU / 24GB
PROMPT
66
tokens/s
GENERATION
6.1
tokens/s
TTFT
19.92
sec
LOCALSCORE
27
Apple M2 4P+4E+8GPU
GPU / 16GB
PROMPT
60
tokens/s
GENERATION
7.0
tokens/s
TTFT
21.43
sec
LOCALSCORE
27
Apple M1 4P+4E+8GPU
GPU / 16GB
PROMPT
53
tokens/s
GENERATION
5.4
tokens/s
TTFT
24.55
sec
LOCALSCORE
23
Apple M1 4P+4E+7GPU
GPU / 16GB
PROMPT
41
tokens/s
GENERATION
4.8
tokens/s
TTFT
30.98
sec
LOCALSCORE
18
AMD Radeon 780M Graphics
GPU / 13GB
PROMPT
57
tokens/s
GENERATION
1.9
tokens/s
TTFT
23.84
sec
LOCALSCORE
17
NVIDIA GeForce RTX 3060 Laptop GPU
GPU / 6GB
PROMPT
83
tokens/s
GENERATION
1.1
tokens/s
TTFT
24.98
sec
LOCALSCORE
15
NVIDIA GeForce GTX 1050 Ti
GPU / 4GB
PROMPT
37
tokens/s
GENERATION
1.2
tokens/s
TTFT
34.58
sec
LOCALSCORE
11