logo
HomeLatest ResultsDownloadAboutBlog
model icon

Qwen2.5 14B Instruct

Q4_K - Medium

14.8Bparams
COMPARE ACCELERATORS

183 accelerators tested

Select Accelerators

NVIDIA RTX PRO 6000 Blackwell Workstation Edition

95GB

NVIDIA GeForce RTX 4090 D

47GB

NVIDIA RTX 6000 Ada Generation

47GB

NVIDIA GeForce RTX 5090

31GB

NVIDIA H100 PCIe

79GB

Qwen2.5 14B Instruct - Q4_K - Medium

LEADERBOARD
NVIDIA RTX PRO 6000 Blackwell Workstation Edition
GPU / 95GB
PROMPT
5099
tokens/s
GENERATION
80.4
tokens/s
TTFT
245
ms
LOCALSCORE
1187
NVIDIA GeForce RTX 4090 D
GPU / 47GB
PROMPT
3905
tokens/s
GENERATION
70.5
tokens/s
TTFT
336
ms
LOCALSCORE
935
NVIDIA RTX 6000 Ada Generation
GPU / 47GB
PROMPT
3552
tokens/s
GENERATION
70.0
tokens/s
TTFT
369
ms
LOCALSCORE
877
NVIDIA GeForce RTX 5090
GPU / 31GB
PROMPT
4193
tokens/s
GENERATION
54.9
tokens/s
TTFT
508
ms
LOCALSCORE
814
NVIDIA H100 PCIe
GPU / 79GB
PROMPT
2973
tokens/s
GENERATION
72.2
tokens/s
TTFT
417
ms
LOCALSCORE
802
NVIDIA GeForce RTX 4080
GPU / 16GB
PROMPT
2665
tokens/s
GENERATION
52.1
tokens/s
TTFT
488
ms
LOCALSCORE
658
NVIDIA GeForce RTX 4090
GPU / 24GB
PROMPT
3033
tokens/s
GENERATION
37.4
tokens/s
TTFT
456
ms
LOCALSCORE
633
NVIDIA GeForce RTX 3090 Ti
GPU / 24GB
PROMPT
2202
tokens/s
GENERATION
64.2
tokens/s
TTFT
575
ms
LOCALSCORE
627
NVIDIA A100-SXM4-80GB
GPU / 79GB
PROMPT
2069
tokens/s
GENERATION
66.3
tokens/s
TTFT
592
ms
LOCALSCORE
614
NVIDIA GeForce RTX 3080 Ti
GPU / 12GB
PROMPT
2090
tokens/s
GENERATION
59.9
tokens/s
TTFT
617
ms
LOCALSCORE
588
NVIDIA GeForce RTX 3090
GPU / 24GB
PROMPT
1990
tokens/s
GENERATION
58.3
tokens/s
TTFT
644
ms
LOCALSCORE
565
NVIDIA RTX 6000 Ada Generation
GPU / 48GB
PROMPT
2936
tokens/s
GENERATION
28.1
tokens/s
TTFT
471
ms
LOCALSCORE
560
NVIDIA GeForce RTX 4070 Ti SUPER
GPU / 16GB
PROMPT
2317
tokens/s
GENERATION
42.7
tokens/s
TTFT
567
ms
LOCALSCORE
557
NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
GPU / 95GB
PROMPT
2839
tokens/s
GENERATION
23.2
tokens/s
TTFT
444
ms
LOCALSCORE
529
NVIDIA GeForce RTX 4070 Ti
GPU / 12GB
PROMPT
2203
tokens/s
GENERATION
40.0
tokens/s
TTFT
616
ms
LOCALSCORE
523
NVIDIA RTX A6000
GPU / 48GB
PROMPT
1883
tokens/s
GENERATION
52.0
tokens/s
TTFT
692
ms
LOCALSCORE
521
NVIDIA L40
GPU / 44GB
PROMPT
2713
tokens/s
GENERATION
24.7
tokens/s
TTFT
513
ms
LOCALSCORE
508
NVIDIA GeForce RTX 4080 SUPER
GPU / 16GB
PROMPT
2424
tokens/s
GENERATION
29.9
tokens/s
TTFT
555
ms
LOCALSCORE
507
NVIDIA H100 80GB HBM3
GPU / 79GB
PROMPT
2609
tokens/s
GENERATION
23.4
tokens/s
TTFT
475
ms
LOCALSCORE
505
NVIDIA GeForce RTX 5070 Ti
GPU / 15GB
PROMPT
2317
tokens/s
GENERATION
47.7
tokens/s
TTFT
859
ms
LOCALSCORE
505
NVIDIA A40
GPU / 44GB
PROMPT
1825
tokens/s
GENERATION
46.2
tokens/s
TTFT
718
ms
LOCALSCORE
490
NVIDIA L40S
GPU / 45GB
PROMPT
2868
tokens/s
GENERATION
19.3
tokens/s
TTFT
495
ms
LOCALSCORE
477
NVIDIA GeForce RTX 5080
GPU / 15GB
PROMPT
2283
tokens/s
GENERATION
23.8
tokens/s
TTFT
562
ms
LOCALSCORE
459
NVIDIA RTX A5000
GPU / 24GB
PROMPT
1532
tokens/s
GENERATION
50.4
tokens/s
TTFT
837
ms
LOCALSCORE
452
NVIDIA H100 NVL
GPU / 93GB
PROMPT
2351
tokens/s
GENERATION
20.5
tokens/s
TTFT
529
ms
LOCALSCORE
450
NVIDIA GeForce RTX 4070
GPU / 12GB
PROMPT
1697
tokens/s
GENERATION
38.8
tokens/s
TTFT
775
ms
LOCALSCORE
440
NVIDIA RTX A4500
GPU / 20GB
PROMPT
1330
tokens/s
GENERATION
43.8
tokens/s
TTFT
979
ms
LOCALSCORE
390
Tesla V100-SXM2-32GB
GPU / 32GB
PROMPT
1161
tokens/s
GENERATION
48.7
tokens/s
TTFT
1.02
sec
LOCALSCORE
381
NVIDIA GeForce RTX 4070 SUPER
GPU / 12GB
PROMPT
1701
tokens/s
GENERATION
24.9
tokens/s
TTFT
786
ms
LOCALSCORE
377
NVIDIA GeForce RTX 5060 Ti
GPU / 15GB
PROMPT
1401
tokens/s
GENERATION
34.0
tokens/s
TTFT
953
ms
LOCALSCORE
368
NVIDIA TITAN RTX
GPU / 23GB
PROMPT
1200
tokens/s
GENERATION
42.8
tokens/s
TTFT
1.05
sec
LOCALSCORE
365
Quadro RTX 8000
GPU / 47GB
PROMPT
1178
tokens/s
GENERATION
42.9
tokens/s
TTFT
1.06
sec
LOCALSCORE
363
NVIDIA GeForce RTX 3080
GPU / 10GB
PROMPT
1451
tokens/s
GENERATION
46.5
tokens/s
TTFT
2.19
sec
LOCALSCORE
358
NVIDIA RTX 3500 Ada Generation Laptop GPU
GPU / 12GB
PROMPT
1337
tokens/s
GENERATION
32.8
tokens/s
TTFT
968
ms
LOCALSCORE
356
NVIDIA RTX 4000 Ada Generation
GPU / 20GB
PROMPT
1417
tokens/s
GENERATION
30.0
tokens/s
TTFT
944
ms
LOCALSCORE
356
NVIDIA GeForce RTX 2080 Ti
GPU / 11GB
PROMPT
1168
tokens/s
GENERATION
39.7
tokens/s
TTFT
1.08
sec
LOCALSCORE
351
NVIDIA TITAN V
GPU / 12GB
PROMPT
962
tokens/s
GENERATION
42.2
tokens/s
TTFT
1.21
sec
LOCALSCORE
323
NVIDIA A100 80GB PCIe
GPU / 79GB
PROMPT
1438
tokens/s
GENERATION
20.8
tokens/s
TTFT
954
ms
LOCALSCORE
315
NVIDIA RTX A4000
GPU / 16GB
PROMPT
1088
tokens/s
GENERATION
34.2
tokens/s
TTFT
1.20
sec
LOCALSCORE
314
NVIDIA GeForce RTX 4060 Ti
GPU / 16GB
PROMPT
1244
tokens/s
GENERATION
26.0
tokens/s
TTFT
1.10
sec
LOCALSCORE
309
NVIDIA GeForce RTX 5070
GPU / 12GB
PROMPT
1264
tokens/s
GENERATION
20.8
tokens/s
TTFT
1.07
sec
LOCALSCORE
290
NVIDIA L4
GPU / 22GB
PROMPT
1095
tokens/s
GENERATION
23.6
tokens/s
TTFT
1.24
sec
LOCALSCORE
275
NVIDIA A10
GPU / 22GB
PROMPT
1133
tokens/s
GENERATION
19.9
tokens/s
TTFT
1.16
sec
LOCALSCORE
269
NVIDIA RTX 4000 SFF Ada Generation
GPU / 20GB
PROMPT
1037
tokens/s
GENERATION
24.9
tokens/s
TTFT
1.34
sec
LOCALSCORE
268
NVIDIA GeForce RTX 3080 Laptop GPU
GPU / 16GB
PROMPT
858
tokens/s
GENERATION
28.2
tokens/s
TTFT
1.46
sec
LOCALSCORE
255
NVIDIA RTX A5000 Laptop GPU
GPU / 16GB
PROMPT
839
tokens/s
GENERATION
27.6
tokens/s
TTFT
1.55
sec
LOCALSCORE
246
NVIDIA GeForce RTX 3060
GPU / 12GB
PROMPT
749
tokens/s
GENERATION
26.6
tokens/s
TTFT
2.07
sec
LOCALSCORE
222
NVIDIA RTX 5000 Ada Generation Laptop GPU
GPU / 16GB
PROMPT
952
tokens/s
GENERATION
16.4
tokens/s
TTFT
1.44
sec
LOCALSCORE
221
Apple M3 Ultra 24P+8E+80GPU
GPU / 256GB
PROMPT
568
tokens/s
GENERATION
36.7
tokens/s
TTFT
2.08
sec
LOCALSCORE
216
Apple M3 Ultra 24P+8E+80GPU
GPU / 512GB
PROMPT
576
tokens/s
GENERATION
35.8
tokens/s
TTFT
2.05
sec
LOCALSCORE
216
NVIDIA RTX 2000 Ada Generation
GPU / 16GB
PROMPT
740
tokens/s
GENERATION
20.7
tokens/s
TTFT
1.81
sec
LOCALSCORE
204
Apple M2 Ultra 16P+8E+76GPU
GPU / 128GB
PROMPT
471
tokens/s
GENERATION
36.6
tokens/s
TTFT
2.48
sec
LOCALSCORE
191
AMD Radeon RX 7900 XTX
GPU / 24GB
PROMPT
485
tokens/s
GENERATION
31.6
tokens/s
TTFT
2.67
sec
LOCALSCORE
179
Apple M3 Ultra 20P+8E+60GPU
GPU / 96GB
PROMPT
445
tokens/s
GENERATION
34.4
tokens/s
TTFT
2.67
sec
LOCALSCORE
179
Radeon RX 7900 XTX
GPU / 24GB
PROMPT
502
tokens/s
GENERATION
27.9
tokens/s
TTFT
2.57
sec
LOCALSCORE
176
NVIDIA RTX A2000 12GB
GPU / 12GB
PROMPT
531
tokens/s
GENERATION
19.7
tokens/s
TTFT
2.41
sec
LOCALSCORE
163
Apple M1 Ultra 16P+4E+64GPU
GPU / 128GB
PROMPT
377
tokens/s
GENERATION
33.3
tokens/s
TTFT
3.10
sec
LOCALSCORE
159
AMD Radeon RX 6900 XT
GPU / 16GB
PROMPT
569
tokens/s
GENERATION
18.5
tokens/s
TTFT
2.63
sec
LOCALSCORE
153
AMD Radeon RX 6800 XT
GPU / 16GB
PROMPT
499
tokens/s
GENERATION
20.3
tokens/s
TTFT
2.90
sec
LOCALSCORE
152
NVIDIA GeForce GTX 1080 Ti
GPU / 11GB
PROMPT
428
tokens/s
GENERATION
20.3
tokens/s
TTFT
2.94
sec
LOCALSCORE
143
Apple M4 Max 12P+4E+40GPU
GPU / 48GB
PROMPT
328
tokens/s
GENERATION
28.0
tokens/s
TTFT
3.60
sec
LOCALSCORE
137
Apple M4 Max 12P+4E+40GPU
GPU / 128GB
PROMPT
325
tokens/s
GENERATION
28.7
tokens/s
TTFT
3.80
sec
LOCALSCORE
135
Tesla T4
GPU / 15GB
PROMPT
450
tokens/s
GENERATION
13.3
tokens/s
TTFT
2.76
sec
LOCALSCORE
130
Apple M1 Ultra 16P+4E+48GPU
GPU / 128GB
PROMPT
290
tokens/s
GENERATION
27.8
tokens/s
TTFT
4.04
sec
LOCALSCORE
126
AMD Radeon RX 7800 XT
GPU / 16GB
PROMPT
302
tokens/s
GENERATION
23.0
tokens/s
TTFT
4.16
sec
LOCALSCORE
119
AMD Radeon RX 7800 XT
GPU / 16GB
PROMPT
309
tokens/s
GENERATION
23.0
tokens/s
TTFT
4.28
sec
LOCALSCORE
118
Apple M4 Max 10P+4E+32GPU
GPU / 36GB
PROMPT
268
tokens/s
GENERATION
25.3
tokens/s
TTFT
4.58
sec
LOCALSCORE
114
Apple M2 Max 8P+4E+38GPU
GPU / 96GB
PROMPT
252
tokens/s
GENERATION
25.2
tokens/s
TTFT
4.80
sec
LOCALSCORE
110
AMD Radeon RX 6800
GPU / 16GB
PROMPT
393
tokens/s
GENERATION
13.5
tokens/s
TTFT
3.73
sec
LOCALSCORE
106
Quadro P5000
GPU / 16GB
PROMPT
303
tokens/s
GENERATION
14.9
tokens/s
TTFT
4.28
sec
LOCALSCORE
102
AMD Radeon RX 6700 XT
GPU / 12GB
PROMPT
316
tokens/s
GENERATION
15.7
tokens/s
TTFT
4.94
sec
LOCALSCORE
100
Apple M3 Max 10P+4E+30GPU
GPU / 96GB
PROMPT
239
tokens/s
GENERATION
20.8
tokens/s
TTFT
5.16
sec
LOCALSCORE
99
Apple M2 Max 8P+4E+38GPU
GPU / 32GB
PROMPT
226
tokens/s
GENERATION
20.6
tokens/s
TTFT
4.98
sec
LOCALSCORE
98
Apple M3 Max 10P+4E+30GPU
GPU / 36GB
PROMPT
226
tokens/s
GENERATION
19.8
tokens/s
TTFT
5.41
sec
LOCALSCORE
94
Tesla P100-SXM2-16GB
GPU / 16GB
PROMPT
204
tokens/s
GENERATION
20.5
tokens/s
TTFT
5.71
sec
LOCALSCORE
90
NVIDIA GeForce RTX 3060 Ti
GPU / 8GB
PROMPT
430
tokens/s
GENERATION
7.5
tokens/s
TTFT
5.55
sec
LOCALSCORE
83
Apple M1 Max 8P+2E+32GPU
GPU / 32GB
PROMPT
189
tokens/s
GENERATION
19.4
tokens/s
TTFT
6.56
sec
LOCALSCORE
82
Apple M4 Pro 10P+4E+20GPU
GPU / 48GB
PROMPT
189
tokens/s
GENERATION
17.9
tokens/s
TTFT
6.66
sec
LOCALSCORE
80
NVIDIA RTX A4500 Laptop GPU
GPU / 15GB
PROMPT
302
tokens/s
GENERATION
7.9
tokens/s
TTFT
4.77
sec
LOCALSCORE
80
Apple M4 Pro 10P+4E+20GPU
GPU / 24GB
PROMPT
186
tokens/s
GENERATION
17.9
tokens/s
TTFT
6.72
sec
LOCALSCORE
79
Apple M1 Max 8P+2E+32GPU
GPU / 64GB
PROMPT
179
tokens/s
GENERATION
17.8
tokens/s
TTFT
6.79
sec
LOCALSCORE
78
Apple M4 Pro 10P+4E+20GPU
GPU / 64GB
PROMPT
179
tokens/s
GENERATION
17.8
tokens/s
TTFT
7.03
sec
LOCALSCORE
77
Apple M3 Max 12P+4E+40GPU
GPU / 64GB
PROMPT
201
tokens/s
GENERATION
13.8
tokens/s
TTFT
6.66
sec
LOCALSCORE
75
Apple M4 Pro 8P+4E+16GPU
GPU / 48GB
PROMPT
161
tokens/s
GENERATION
16.8
tokens/s
TTFT
7.84
sec
LOCALSCORE
70
Apple M4 Pro 8P+4E+16GPU
GPU / 24GB
PROMPT
153
tokens/s
GENERATION
16.6
tokens/s
TTFT
8.25
sec
LOCALSCORE
67
Apple M2 Max 8P+4E+30GPU
GPU / 64GB
PROMPT
149
tokens/s
GENERATION
14.5
tokens/s
TTFT
7.68
sec
LOCALSCORE
66
NVIDIA GeForce RTX 3070
GPU / 8GB
PROMPT
376
tokens/s
GENERATION
5.1
tokens/s
TTFT
8.07
sec
LOCALSCORE
62
Apple M1 Max 8P+2E+24GPU
GPU / 64GB
PROMPT
136
tokens/s
GENERATION
14.6
tokens/s
TTFT
9.14
sec
LOCALSCORE
61
NVIDIA GeForce RTX 2080
GPU / 8GB
PROMPT
324
tokens/s
GENERATION
5.3
tokens/s
TTFT
7.87
sec
LOCALSCORE
60
Apple M2 Pro 8P+4E+19GPU
GPU / 32GB
PROMPT
137
tokens/s
GENERATION
14.1
tokens/s
TTFT
9.10
sec
LOCALSCORE
60
Apple M3 Pro 6P+6E+18GPU
GPU / 36GB
PROMPT
150
tokens/s
GENERATION
12.1
tokens/s
TTFT
8.56
sec
LOCALSCORE
60
Apple M3 Pro 6P+6E+18GPU
GPU / 18GB
PROMPT
144
tokens/s
GENERATION
11.6
tokens/s
TTFT
8.97
sec
LOCALSCORE
57
Apple M2 Pro 6P+4E+16GPU
GPU / 16GB
PROMPT
119
tokens/s
GENERATION
13.3
tokens/s
TTFT
10.66
sec
LOCALSCORE
53
NVIDIA GeForce RTX 4060 Ti
GPU / 8GB
PROMPT
327
tokens/s
GENERATION
3.5
tokens/s
TTFT
7.99
sec
LOCALSCORE
52
Apple M3 Pro 5P+6E+14GPU
GPU / 36GB
PROMPT
120
tokens/s
GENERATION
12.1
tokens/s
TTFT
10.65
sec
LOCALSCORE
51
Apple M3 Pro 5P+6E+14GPU
GPU / 18GB
PROMPT
117
tokens/s
GENERATION
11.9
tokens/s
TTFT
10.79
sec
LOCALSCORE
51
Apple M1 Pro 8P+2E+16GPU
GPU / 16GB
PROMPT
107
tokens/s
GENERATION
11.9
tokens/s
TTFT
11.84
sec
LOCALSCORE
48
Apple M1 Pro 8P+2E+16GPU
GPU / 32GB
PROMPT
104
tokens/s
GENERATION
11.5
tokens/s
TTFT
12.11
sec
LOCALSCORE
46
Apple M1 Pro 6P+2E+14GPU
GPU / 16GB
PROMPT
94
tokens/s
GENERATION
10.9
tokens/s
TTFT
13.34
sec
LOCALSCORE
42
Apple M4 4P+6E+10GPU
GPU / 16GB
PROMPT
79
tokens/s
GENERATION
8.0
tokens/s
TTFT
16.91
sec
LOCALSCORE
34
NVIDIA RTX PRO 2000 Blackwell Generation Laptop GPU
GPU / 8GB
PROMPT
230
tokens/s
GENERATION
2.1
tokens/s
TTFT
14.47
sec
LOCALSCORE
32
Apple M4 4P+6E+10GPU
GPU / 32GB
PROMPT
71
tokens/s
GENERATION
8.1
tokens/s
TTFT
17.08
sec
LOCALSCORE
32
Apple M2 4P+4E+10GPU
GPU / 24GB
PROMPT
70
tokens/s
GENERATION
7.3
tokens/s
TTFT
18.03
sec
LOCALSCORE
31
AMD Radeon Graphics
GPU / 32GB
PROMPT
53
tokens/s
GENERATION
14.4
tokens/s
TTFT
27.08
sec
LOCALSCORE
30
AMD Instinct MI50/MI60
GPU / 16GB
PROMPT
48
tokens/s
GENERATION
12.9
tokens/s
TTFT
30.10
sec
LOCALSCORE
27
Apple M3 4P+4E+10GPU
GPU / 24GB
PROMPT
66
tokens/s
GENERATION
6.2
tokens/s
TTFT
20.23
sec
LOCALSCORE
27
Apple M1 4P+4E+8GPU
GPU / 16GB
PROMPT
53
tokens/s
GENERATION
5.4
tokens/s
TTFT
24.55
sec
LOCALSCORE
23
Apple M1 4P+4E+7GPU
GPU / 16GB
PROMPT
41
tokens/s
GENERATION
4.8
tokens/s
TTFT
31.76
sec
LOCALSCORE
18