logo
HomeLatest ResultsDownloadAboutBlog
model icon

Qwen2.5 14B Instruct

Q4_K - Medium

14.8Bparams
COMPARE ACCELERATORS

259 accelerators tested

Select Accelerators

Apple M3 4P+4E+10GPU

16GB

NVIDIA RTX PRO 6000 Blackwell Workstation Edition

95GB

NVIDIA GeForce RTX 4090

23GB

NVIDIA GeForce RTX 4090 D

47GB

NVIDIA RTX 6000 Ada Generation

47GB

Qwen2.5 14B Instruct - Q4_K - Medium

LEADERBOARD
Apple M3 4P+4E+10GPU
GPU / 16GB
PROMPT
9787
tokens/s
GENERATION
1564
tokens/s
TTFT
120
ms
LOCALSCORE
5027
NVIDIA RTX PRO 6000 Blackwell Workstation Edition
GPU / 95GB
PROMPT
5255
tokens/s
GENERATION
81.9
tokens/s
TTFT
240
ms
LOCALSCORE
1215
NVIDIA GeForce RTX 4090
GPU / 23GB
PROMPT
4208
tokens/s
GENERATION
68.8
tokens/s
TTFT
315
ms
LOCALSCORE
972
NVIDIA GeForce RTX 4090 D
GPU / 47GB
PROMPT
3905
tokens/s
GENERATION
70.5
tokens/s
TTFT
336
ms
LOCALSCORE
935
NVIDIA RTX 6000 Ada Generation
GPU / 47GB
PROMPT
3552
tokens/s
GENERATION
70.0
tokens/s
TTFT
369
ms
LOCALSCORE
877
NVIDIA GeForce RTX 4090
GPU / 22GB
PROMPT
4028
tokens/s
GENERATION
51.8
tokens/s
TTFT
334
ms
LOCALSCORE
854
NVIDIA H100 PCIe
GPU / 79GB
PROMPT
2973
tokens/s
GENERATION
72.2
tokens/s
TTFT
417
ms
LOCALSCORE
802
NVIDIA GeForce RTX 4090
GPU / 24GB
PROMPT
3296
tokens/s
GENERATION
45.3
tokens/s
TTFT
423
ms
LOCALSCORE
714
NVIDIA GeForce RTX 5090
GPU / 31GB
PROMPT
3678
tokens/s
GENERATION
45.5
tokens/s
TTFT
536
ms
LOCALSCORE
708
NVIDIA GeForce RTX 5090
GPU / 32GB
PROMPT
4295
tokens/s
GENERATION
49.5
tokens/s
TTFT
631
ms
LOCALSCORE
696
NVIDIA GeForce RTX 3090 Ti
GPU / 24GB
PROMPT
2202
tokens/s
GENERATION
64.2
tokens/s
TTFT
575
ms
LOCALSCORE
627
NVIDIA A100-SXM4-80GB
GPU / 79GB
PROMPT
2069
tokens/s
GENERATION
66.3
tokens/s
TTFT
592
ms
LOCALSCORE
614
NVIDIA GeForce RTX 4080 SUPER
GPU / 16GB
PROMPT
2681
tokens/s
GENERATION
43.6
tokens/s
TTFT
503
ms
LOCALSCORE
614
NVIDIA RTX 6000 Ada Generation
GPU / 48GB
PROMPT
2936
tokens/s
GENERATION
28.1
tokens/s
TTFT
471
ms
LOCALSCORE
560
NVIDIA GeForce RTX 3090
GPU / 24GB
PROMPT
1955
tokens/s
GENERATION
56.3
tokens/s
TTFT
655
ms
LOCALSCORE
552
NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
GPU / 95GB
PROMPT
2913
tokens/s
GENERATION
24.1
tokens/s
TTFT
434
ms
LOCALSCORE
545
NVIDIA GeForce RTX 4070 Ti SUPER
GPU / 16GB
PROMPT
2271
tokens/s
GENERATION
41.0
tokens/s
TTFT
580
ms
LOCALSCORE
543
NVIDIA GeForce RTX 4070 Ti
GPU / 12GB
PROMPT
2203
tokens/s
GENERATION
40.0
tokens/s
TTFT
616
ms
LOCALSCORE
523
NVIDIA RTX A6000
GPU / 48GB
PROMPT
1883
tokens/s
GENERATION
52.0
tokens/s
TTFT
692
ms
LOCALSCORE
521
NVIDIA L40
GPU / 44GB
PROMPT
2713
tokens/s
GENERATION
24.7
tokens/s
TTFT
513
ms
LOCALSCORE
508
NVIDIA H100 80GB HBM3
GPU / 79GB
PROMPT
2609
tokens/s
GENERATION
23.4
tokens/s
TTFT
475
ms
LOCALSCORE
505
NVIDIA GeForce RTX 5070 Ti
GPU / 15GB
PROMPT
2317
tokens/s
GENERATION
47.7
tokens/s
TTFT
859
ms
LOCALSCORE
505
NVIDIA A40
GPU / 44GB
PROMPT
1825
tokens/s
GENERATION
46.2
tokens/s
TTFT
718
ms
LOCALSCORE
490
NVIDIA L40S
GPU / 45GB
PROMPT
2868
tokens/s
GENERATION
19.3
tokens/s
TTFT
495
ms
LOCALSCORE
477
NVIDIA GeForce RTX 3080 Ti
GPU / 12GB
PROMPT
1797
tokens/s
GENERATION
43.7
tokens/s
TTFT
738
ms
LOCALSCORE
467
NVIDIA GeForce RTX 5080
GPU / 15GB
PROMPT
2283
tokens/s
GENERATION
23.8
tokens/s
TTFT
562
ms
LOCALSCORE
459
NVIDIA RTX A5000
GPU / 24GB
PROMPT
1532
tokens/s
GENERATION
50.4
tokens/s
TTFT
837
ms
LOCALSCORE
452
NVIDIA H100 NVL
GPU / 93GB
PROMPT
2354
tokens/s
GENERATION
20.5
tokens/s
TTFT
528
ms
LOCALSCORE
450
NVIDIA RTX PRO 6000 Blackwell Server Edition
GPU / 95GB
PROMPT
2581
tokens/s
GENERATION
17.9
tokens/s
TTFT
637
ms
LOCALSCORE
438
NVIDIA GeForce RTX 4070
GPU / 12GB
PROMPT
1692
tokens/s
GENERATION
37.9
tokens/s
TTFT
779
ms
LOCALSCORE
435
NVIDIA GeForce RTX 4080
GPU / 16GB
PROMPT
2057
tokens/s
GENERATION
26.4
tokens/s
TTFT
670
ms
LOCALSCORE
429
NVIDIA GeForce RTX 5070 Ti
GPU / 16GB
PROMPT
1740
tokens/s
GENERATION
28.3
tokens/s
TTFT
838
ms
LOCALSCORE
406
NVIDIA RTX A4500
GPU / 20GB
PROMPT
1330
tokens/s
GENERATION
43.8
tokens/s
TTFT
979
ms
LOCALSCORE
390
Tesla V100-SXM2-32GB
GPU / 32GB
PROMPT
1161
tokens/s
GENERATION
48.7
tokens/s
TTFT
1.02
sec
LOCALSCORE
381
NVIDIA GeForce RTX 4070 SUPER
GPU / 12GB
PROMPT
1701
tokens/s
GENERATION
24.9
tokens/s
TTFT
786
ms
LOCALSCORE
377
NVIDIA TITAN RTX
GPU / 23GB
PROMPT
1200
tokens/s
GENERATION
42.8
tokens/s
TTFT
1.05
sec
LOCALSCORE
365
Quadro RTX 8000
GPU / 47GB
PROMPT
1178
tokens/s
GENERATION
42.9
tokens/s
TTFT
1.06
sec
LOCALSCORE
363
NVIDIA GeForce RTX 3080
GPU / 10GB
PROMPT
1451
tokens/s
GENERATION
46.5
tokens/s
TTFT
2.19
sec
LOCALSCORE
358
NVIDIA RTX 4000 Ada Generation
GPU / 20GB
PROMPT
1417
tokens/s
GENERATION
30.0
tokens/s
TTFT
944
ms
LOCALSCORE
356
NVIDIA GeForce RTX 2080 Ti
GPU / 11GB
PROMPT
1140
tokens/s
GENERATION
39.5
tokens/s
TTFT
1.09
sec
LOCALSCORE
346
NVIDIA GeForce RTX 5060 Ti
GPU / 15GB
PROMPT
1364
tokens/s
GENERATION
33.5
tokens/s
TTFT
1.26
sec
LOCALSCORE
345
NVIDIA TITAN V
GPU / 12GB
PROMPT
962
tokens/s
GENERATION
42.2
tokens/s
TTFT
1.21
sec
LOCALSCORE
323
NVIDIA A100 80GB PCIe
GPU / 79GB
PROMPT
1438
tokens/s
GENERATION
20.8
tokens/s
TTFT
954
ms
LOCALSCORE
315
NVIDIA RTX 3500 Ada Generation Laptop GPU
GPU / 12GB
PROMPT
1298
tokens/s
GENERATION
25.1
tokens/s
TTFT
1.02
sec
LOCALSCORE
315
NVIDIA GeForce RTX 4060 Ti
GPU / 16GB
PROMPT
1244
tokens/s
GENERATION
26.0
tokens/s
TTFT
1.10
sec
LOCALSCORE
309
NVIDIA RTX A4000
GPU / 16GB
PROMPT
1070
tokens/s
GENERATION
32.3
tokens/s
TTFT
1.21
sec
LOCALSCORE
305
NVIDIA GeForce RTX 5060 Ti
GPU / 16GB
PROMPT
1284
tokens/s
GENERATION
22.6
tokens/s
TTFT
1.03
sec
LOCALSCORE
304
NVIDIA GeForce RTX 5070
GPU / 12GB
PROMPT
1264
tokens/s
GENERATION
20.8
tokens/s
TTFT
1.07
sec
LOCALSCORE
290
NVIDIA L4
GPU / 22GB
PROMPT
1095
tokens/s
GENERATION
23.6
tokens/s
TTFT
1.24
sec
LOCALSCORE
275
NVIDIA RTX 4000 SFF Ada Generation
GPU / 20GB
PROMPT
1051
tokens/s
GENERATION
24.7
tokens/s
TTFT
1.30
sec
LOCALSCORE
271
NVIDIA A10
GPU / 22GB
PROMPT
1133
tokens/s
GENERATION
19.9
tokens/s
TTFT
1.16
sec
LOCALSCORE
269
NVIDIA GeForce RTX 3080 Laptop GPU
GPU / 16GB
PROMPT
858
tokens/s
GENERATION
28.2
tokens/s
TTFT
1.46
sec
LOCALSCORE
255
NVIDIA GB10
GPU / 120GB
PROMPT
1112
tokens/s
GENERATION
18.2
tokens/s
TTFT
1.22
sec
LOCALSCORE
255
NVIDIA RTX A5000 Laptop GPU
GPU / 16GB
PROMPT
839
tokens/s
GENERATION
27.6
tokens/s
TTFT
1.55
sec
LOCALSCORE
246
NVIDIA GeForce RTX 3060
GPU / 12GB
PROMPT
751
tokens/s
GENERATION
26.7
tokens/s
TTFT
2.03
sec
LOCALSCORE
223
NVIDIA RTX 5000 Ada Generation Laptop GPU
GPU / 16GB
PROMPT
952
tokens/s
GENERATION
16.4
tokens/s
TTFT
1.44
sec
LOCALSCORE
221
Apple M3 Ultra 24P+8E+80GPU
GPU / 512GB
PROMPT
577
tokens/s
GENERATION
35.8
tokens/s
TTFT
2.05
sec
LOCALSCORE
216
Apple M3 Ultra 24P+8E+80GPU
GPU / 256GB
PROMPT
568
tokens/s
GENERATION
36.7
tokens/s
TTFT
2.08
sec
LOCALSCORE
216
NVIDIA RTX 2000 Ada Generation
GPU / 16GB
PROMPT
740
tokens/s
GENERATION
20.7
tokens/s
TTFT
1.81
sec
LOCALSCORE
204
Apple M2 Ultra 16P+8E+76GPU
GPU / 128GB
PROMPT
471
tokens/s
GENERATION
36.6
tokens/s
TTFT
2.48
sec
LOCALSCORE
191
NVIDIA GeForce RTX 2060
GPU / 12GB
PROMPT
595
tokens/s
GENERATION
22.8
tokens/s
TTFT
2.16
sec
LOCALSCORE
184
AMD Radeon RX 7900 XTX
GPU / 24GB
PROMPT
485
tokens/s
GENERATION
31.6
tokens/s
TTFT
2.67
sec
LOCALSCORE
179
Apple M3 Ultra 20P+8E+60GPU
GPU / 96GB
PROMPT
445
tokens/s
GENERATION
34.4
tokens/s
TTFT
2.67
sec
LOCALSCORE
179
Radeon RX 7900 XTX
GPU / 24GB
PROMPT
502
tokens/s
GENERATION
27.9
tokens/s
TTFT
2.57
sec
LOCALSCORE
176
Apple M2 Ultra 16P+8E+60GPU
GPU / 64GB
PROMPT
381
tokens/s
GENERATION
34.2
tokens/s
TTFT
3.06
sec
LOCALSCORE
162
Apple M1 Ultra 16P+4E+64GPU
GPU / 128GB
PROMPT
372
tokens/s
GENERATION
32.4
tokens/s
TTFT
3.13
sec
LOCALSCORE
157
NVIDIA RTX A2000 12GB
GPU / 12GB
PROMPT
502
tokens/s
GENERATION
19.3
tokens/s
TTFT
2.53
sec
LOCALSCORE
157
AMD Radeon RX 6900 XT
GPU / 16GB
PROMPT
569
tokens/s
GENERATION
18.5
tokens/s
TTFT
2.63
sec
LOCALSCORE
153
AMD Radeon RX 6800 XT
GPU / 16GB
PROMPT
499
tokens/s
GENERATION
20.3
tokens/s
TTFT
2.90
sec
LOCALSCORE
152
Apple M4 Max 12P+4E+40GPU
GPU / 48GB
PROMPT
347
tokens/s
GENERATION
30.1
tokens/s
TTFT
3.49
sec
LOCALSCORE
144
NVIDIA GeForce GTX 1080 Ti
GPU / 11GB
PROMPT
419
tokens/s
GENERATION
20.2
tokens/s
TTFT
3.05
sec
LOCALSCORE
141
Apple M4 Max 12P+4E+40GPU
GPU / 128GB
PROMPT
328
tokens/s
GENERATION
28.8
tokens/s
TTFT
3.76
sec
LOCALSCORE
136
Tesla T4
GPU / 15GB
PROMPT
450
tokens/s
GENERATION
13.3
tokens/s
TTFT
2.76
sec
LOCALSCORE
130
Apple M4 Max 12P+4E+40GPU
GPU / 64GB
PROMPT
306
tokens/s
GENERATION
27.7
tokens/s
TTFT
4.07
sec
LOCALSCORE
128
Apple M1 Ultra 16P+4E+48GPU
GPU / 128GB
PROMPT
290
tokens/s
GENERATION
27.8
tokens/s
TTFT
4.04
sec
LOCALSCORE
126
Apple M3 Max 12P+4E+40GPU
GPU / 128GB
PROMPT
302
tokens/s
GENERATION
25.5
tokens/s
TTFT
4.07
sec
LOCALSCORE
124
AMD Radeon RX 7800 XT
GPU / 16GB
PROMPT
302
tokens/s
GENERATION
23.0
tokens/s
TTFT
4.16
sec
LOCALSCORE
119
AMD Radeon RX 7800 XT
GPU / 16GB
PROMPT
306
tokens/s
GENERATION
22.9
tokens/s
TTFT
4.29
sec
LOCALSCORE
118
Tesla P40
GPU / 24GB
PROMPT
339
tokens/s
GENERATION
15.7
tokens/s
TTFT
3.50
sec
LOCALSCORE
115
AMD Radeon RX 6800
GPU / 16GB
PROMPT
399
tokens/s
GENERATION
14.8
tokens/s
TTFT
3.68
sec
LOCALSCORE
112
Apple M4 Max 10P+4E+32GPU
GPU / 36GB
PROMPT
267
tokens/s
GENERATION
24.0
tokens/s
TTFT
4.51
sec
LOCALSCORE
112
Apple M2 Max 8P+4E+38GPU
GPU / 96GB
PROMPT
252
tokens/s
GENERATION
25.2
tokens/s
TTFT
4.80
sec
LOCALSCORE
110
Apple M2 Max 8P+4E+38GPU
GPU / 64GB
PROMPT
246
tokens/s
GENERATION
23.8
tokens/s
TTFT
4.92
sec
LOCALSCORE
106
Quadro P5000
GPU / 16GB
PROMPT
303
tokens/s
GENERATION
14.9
tokens/s
TTFT
4.28
sec
LOCALSCORE
102
AMD Radeon RX 6700 XT
GPU / 12GB
PROMPT
316
tokens/s
GENERATION
15.7
tokens/s
TTFT
4.94
sec
LOCALSCORE
100
Apple M3 Max 10P+4E+30GPU
GPU / 96GB
PROMPT
239
tokens/s
GENERATION
20.8
tokens/s
TTFT
5.16
sec
LOCALSCORE
99
Apple M2 Max 8P+4E+38GPU
GPU / 32GB
PROMPT
226
tokens/s
GENERATION
20.6
tokens/s
TTFT
4.98
sec
LOCALSCORE
98
Apple M3 Max 10P+4E+30GPU
GPU / 36GB
PROMPT
226
tokens/s
GENERATION
19.8
tokens/s
TTFT
5.41
sec
LOCALSCORE
94
Tesla P100-SXM2-16GB
GPU / 16GB
PROMPT
204
tokens/s
GENERATION
20.5
tokens/s
TTFT
5.71
sec
LOCALSCORE
90
AMD Radeon RX 6950 XT
GPU / 16GB
PROMPT
549
tokens/s
GENERATION
3.3
tokens/s
TTFT
2.85
sec
LOCALSCORE
86
Apple M1 Max 8P+2E+32GPU
GPU / 32GB
PROMPT
194
tokens/s
GENERATION
19.9
tokens/s
TTFT
6.36
sec
LOCALSCORE
85
NVIDIA GeForce RTX 3060 Ti
GPU / 8GB
PROMPT
430
tokens/s
GENERATION
7.5
tokens/s
TTFT
5.55
sec
LOCALSCORE
83
Apple M4 Pro 10P+4E+20GPU
GPU / 24GB
PROMPT
190
tokens/s
GENERATION
18.0
tokens/s
TTFT
6.63
sec
LOCALSCORE
80
Apple M4 Pro 10P+4E+20GPU
GPU / 48GB
PROMPT
189
tokens/s
GENERATION
18.0
tokens/s
TTFT
6.64
sec
LOCALSCORE
80
NVIDIA RTX A4500 Laptop GPU
GPU / 15GB
PROMPT
302
tokens/s
GENERATION
7.9
tokens/s
TTFT
4.77
sec
LOCALSCORE
80
Apple M1 Max 8P+2E+32GPU
GPU / 64GB
PROMPT
181
tokens/s
GENERATION
18.5
tokens/s
TTFT
7.28
sec
LOCALSCORE
79
Apple M4 Pro 10P+4E+20GPU
GPU / 64GB
PROMPT
179
tokens/s
GENERATION
17.8
tokens/s
TTFT
7.05
sec
LOCALSCORE
77
Apple M3 Max 12P+4E+40GPU
GPU / 64GB
PROMPT
201
tokens/s
GENERATION
13.8
tokens/s
TTFT
6.66
sec
LOCALSCORE
75
Apple M4 Pro 8P+4E+16GPU
GPU / 48GB
PROMPT
161
tokens/s
GENERATION
16.8
tokens/s
TTFT
7.84
sec
LOCALSCORE
70
Apple M1 Max 8P+2E+24GPU
GPU / 32GB
PROMPT
156
tokens/s
GENERATION
17.4
tokens/s
TTFT
7.86
sec
LOCALSCORE
70
Apple M4 Pro 8P+4E+16GPU
GPU / 64GB
PROMPT
151
tokens/s
GENERATION
16.1
tokens/s
TTFT
8.46
sec
LOCALSCORE
66
Apple M2 Max 8P+4E+30GPU
GPU / 64GB
PROMPT
149
tokens/s
GENERATION
14.5
tokens/s
TTFT
7.68
sec
LOCALSCORE
66
Apple M4 Pro 8P+4E+16GPU
GPU / 24GB
PROMPT
144
tokens/s
GENERATION
15.2
tokens/s
TTFT
8.90
sec
LOCALSCORE
63
Apple M1 Max 8P+2E+24GPU
GPU / 64GB
PROMPT
140
tokens/s
GENERATION
15.1
tokens/s
TTFT
8.83
sec
LOCALSCORE
63
NVIDIA GeForce RTX 3070
GPU / 8GB
PROMPT
376
tokens/s
GENERATION
5.1
tokens/s
TTFT
8.07
sec
LOCALSCORE
62
NVIDIA GeForce RTX 2080
GPU / 8GB
PROMPT
324
tokens/s
GENERATION
5.3
tokens/s
TTFT
7.87
sec
LOCALSCORE
60
Apple M2 Pro 8P+4E+19GPU
GPU / 32GB
PROMPT
137
tokens/s
GENERATION
14.1
tokens/s
TTFT
9.10
sec
LOCALSCORE
60
NVIDIA L40S-8Q
GPU / 8GB
PROMPT
343
tokens/s
GENERATION
3.5
tokens/s
TTFT
5.74
sec
LOCALSCORE
59
Apple M3 Pro 6P+6E+18GPU
GPU / 36GB
PROMPT
147
tokens/s
GENERATION
12.0
tokens/s
TTFT
8.73
sec
LOCALSCORE
59
Apple M3 Pro 6P+6E+18GPU
GPU / 18GB
PROMPT
145
tokens/s
GENERATION
11.6
tokens/s
TTFT
8.88
sec
LOCALSCORE
57
NVIDIA GeForce RTX 2080 SUPER
GPU / 8GB
PROMPT
294
tokens/s
GENERATION
4.5
tokens/s
TTFT
8.27
sec
LOCALSCORE
54
Apple M2 Pro 6P+4E+16GPU
GPU / 16GB
PROMPT
119
tokens/s
GENERATION
13.4
tokens/s
TTFT
10.62
sec
LOCALSCORE
53
NVIDIA GeForce RTX 4060 Ti
GPU / 8GB
PROMPT
327
tokens/s
GENERATION
3.5
tokens/s
TTFT
7.99
sec
LOCALSCORE
52
Apple M3 Pro 5P+6E+14GPU
GPU / 36GB
PROMPT
120
tokens/s
GENERATION
12.1
tokens/s
TTFT
10.65
sec
LOCALSCORE
51
Apple M3 Pro 5P+6E+14GPU
GPU / 18GB
PROMPT
117
tokens/s
GENERATION
11.9
tokens/s
TTFT
10.79
sec
LOCALSCORE
51
NVIDIA GeForce RTX 4070 Laptop GPU
GPU / 8GB
PROMPT
289
tokens/s
GENERATION
3.8
tokens/s
TTFT
8.84
sec
LOCALSCORE
50
Apple M1 Pro 8P+2E+16GPU
GPU / 16GB
PROMPT
107
tokens/s
GENERATION
11.9
tokens/s
TTFT
11.84
sec
LOCALSCORE
48
Apple M1 Pro 8P+2E+16GPU
GPU / 32GB
PROMPT
104
tokens/s
GENERATION
11.6
tokens/s
TTFT
12.08
sec
LOCALSCORE
46
Apple M1 Pro 6P+2E+14GPU
GPU / 16GB
PROMPT
92
tokens/s
GENERATION
10.8
tokens/s
TTFT
13.49
sec
LOCALSCORE
42
Apple M1 Pro 8P+2E+14GPU
GPU / 32GB
PROMPT
89
tokens/s
GENERATION
10.4
tokens/s
TTFT
14.17
sec
LOCALSCORE
40
Apple M4 4P+6E+10GPU
GPU / 24GB
PROMPT
93
tokens/s
GENERATION
9.2
tokens/s
TTFT
13.71
sec
LOCALSCORE
40
Apple M4 4P+6E+10GPU
GPU / 16GB
PROMPT
85
tokens/s
GENERATION
8.8
tokens/s
TTFT
15.38
sec
LOCALSCORE
37
Apple M4 4P+6E+10GPU
GPU / 32GB
PROMPT
79
tokens/s
GENERATION
8.6
tokens/s
TTFT
15.82
sec
LOCALSCORE
35
Apple M2 4P+4E+10GPU
GPU / 16GB
PROMPT
75
tokens/s
GENERATION
8.1
tokens/s
TTFT
17.30
sec
LOCALSCORE
33
NVIDIA RTX PRO 2000 Blackwell Generation Laptop GPU
GPU / 8GB
PROMPT
230
tokens/s
GENERATION
2.1
tokens/s
TTFT
14.47
sec
LOCALSCORE
32
Apple M2 4P+4E+10GPU
GPU / 24GB
PROMPT
70
tokens/s
GENERATION
7.3
tokens/s
TTFT
18.03
sec
LOCALSCORE
31
AMD Radeon Graphics
GPU / 32GB
PROMPT
51
tokens/s
GENERATION
14.1
tokens/s
TTFT
28.62
sec
LOCALSCORE
29
Apple M4 4P+6E+8GPU
GPU / 16GB
PROMPT
61
tokens/s
GENERATION
7.2
tokens/s
TTFT
20.48
sec
LOCALSCORE
28
AMD Instinct MI50/MI60
GPU / 16GB
PROMPT
48
tokens/s
GENERATION
12.9
tokens/s
TTFT
30.10
sec
LOCALSCORE
27
Apple M3 4P+4E+10GPU
GPU / 24GB
PROMPT
66
tokens/s
GENERATION
6.1
tokens/s
TTFT
19.92
sec
LOCALSCORE
27
Apple M2 4P+4E+8GPU
GPU / 16GB
PROMPT
60
tokens/s
GENERATION
7.0
tokens/s
TTFT
21.43
sec
LOCALSCORE
27
Apple M1 4P+4E+8GPU
GPU / 16GB
PROMPT
53
tokens/s
GENERATION
5.4
tokens/s
TTFT
24.55
sec
LOCALSCORE
23
Apple M1 4P+4E+7GPU
GPU / 16GB
PROMPT
41
tokens/s
GENERATION
4.8
tokens/s
TTFT
31.76
sec
LOCALSCORE
18
NVIDIA GeForce RTX 3060 Laptop GPU
GPU / 6GB
PROMPT
83
tokens/s
GENERATION
1.1
tokens/s
TTFT
24.98
sec
LOCALSCORE
15