logo
HomeLatest ResultsDownloadAboutBlog
model icon

Llama 3.2 1B Instruct

Q4_K - Medium

1.5Bparams
COMPARE ACCELERATORS

446 accelerators tested

Select Accelerators

NVIDIA RTX 6000 Ada Generation

47GB

NVIDIA GeForce RTX 4090

23GB

NVIDIA GeForce RTX 4090 D

47GB

NVIDIA L40S

44GB

NVIDIA RTX PRO 6000 Blackwell Workstation Edition

95GB

Llama 3.2 1B Instruct - Q4_K - Medium

LEADERBOARD
NVIDIA RTX 6000 Ada Generation
GPU / 47GB
PROMPT
27281
tokens/s
GENERATION
411
tokens/s
TTFT
53
ms
LOCALSCORE
5948
NVIDIA GeForce RTX 4090
GPU / 23GB
PROMPT
25538
tokens/s
GENERATION
382
tokens/s
TTFT
57
ms
LOCALSCORE
5562
NVIDIA GeForce RTX 4090 D
GPU / 47GB
PROMPT
24549
tokens/s
GENERATION
359
tokens/s
TTFT
57
ms
LOCALSCORE
5380
NVIDIA L40S
GPU / 44GB
PROMPT
24736
tokens/s
GENERATION
361
tokens/s
TTFT
60
ms
LOCALSCORE
5287
NVIDIA RTX PRO 6000 Blackwell Workstation Edition
GPU / 95GB
PROMPT
25401
tokens/s
GENERATION
244
tokens/s
TTFT
55
ms
LOCALSCORE
4842
NVIDIA H100 PCIe
GPU / 79GB
PROMPT
18590
tokens/s
GENERATION
335
tokens/s
TTFT
68
ms
LOCALSCORE
4512
NVIDIA GeForce RTX 3090 Ti
GPU / 24GB
PROMPT
15120
tokens/s
GENERATION
354
tokens/s
TTFT
90
ms
LOCALSCORE
3908
NVIDIA A100-SXM4-80GB
GPU / 79GB
PROMPT
15370
tokens/s
GENERATION
308
tokens/s
TTFT
80
ms
LOCALSCORE
3892
NVIDIA GeForce RTX 3080 Ti
GPU / 12GB
PROMPT
14352
tokens/s
GENERATION
326
tokens/s
TTFT
93
ms
LOCALSCORE
3688
NVIDIA RTX A6000
GPU / 48GB
PROMPT
13191
tokens/s
GENERATION
315
tokens/s
TTFT
102
ms
LOCALSCORE
3438
NVIDIA RTX 6000 Ada Generation
GPU / 48GB
PROMPT
19620
tokens/s
GENERATION
131
tokens/s
TTFT
68
ms
LOCALSCORE
3350
NVIDIA GeForce RTX 5090
GPU / 31GB
PROMPT
20518
tokens/s
GENERATION
175
tokens/s
TTFT
325
ms
LOCALSCORE
3248
NVIDIA RTX A5000
GPU / 24GB
PROMPT
12138
tokens/s
GENERATION
310
tokens/s
TTFT
110
ms
LOCALSCORE
3242
NVIDIA GeForce RTX 4070 Ti SUPER
GPU / 16GB
PROMPT
15287
tokens/s
GENERATION
206
tokens/s
TTFT
93
ms
LOCALSCORE
3211
NVIDIA GeForce RTX 4070
GPU / 12GB
PROMPT
12717
tokens/s
GENERATION
283
tokens/s
TTFT
112
ms
LOCALSCORE
3179
NVIDIA GeForce RTX 3090
GPU / 24GB
PROMPT
13233
tokens/s
GENERATION
249
tokens/s
TTFT
102
ms
LOCALSCORE
3152
NVIDIA GeForce RTX 4080
GPU / 16GB
PROMPT
14496
tokens/s
GENERATION
241
tokens/s
TTFT
130
ms
LOCALSCORE
3133
NVIDIA A40
GPU / 44GB
PROMPT
12156
tokens/s
GENERATION
284
tokens/s
TTFT
113
ms
LOCALSCORE
3126
NVIDIA L40S
GPU / 45GB
PROMPT
18255
tokens/s
GENERATION
121
tokens/s
TTFT
75
ms
LOCALSCORE
3088
NVIDIA GeForce RTX 4080 SUPER
GPU / 16GB
PROMPT
16736
tokens/s
GENERATION
141
tokens/s
TTFT
81
ms
LOCALSCORE
3077
NVIDIA GeForce RTX 4090
GPU / 24GB
PROMPT
16850
tokens/s
GENERATION
159
tokens/s
TTFT
116
ms
LOCALSCORE
3038
NVIDIA L40
GPU / 44GB
PROMPT
17686
tokens/s
GENERATION
118
tokens/s
TTFT
77
ms
LOCALSCORE
3000
NVIDIA RTX A4500
GPU / 20GB
PROMPT
10328
tokens/s
GENERATION
285
tokens/s
TTFT
131
ms
LOCALSCORE
2820
NVIDIA GeForce RTX 3070 Ti
GPU / 8GB
PROMPT
10074
tokens/s
GENERATION
297
tokens/s
TTFT
137
ms
LOCALSCORE
2795
NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
GPU / 95GB
PROMPT
16184
tokens/s
GENERATION
106
tokens/s
TTFT
80
ms
LOCALSCORE
2780
NVIDIA GeForce RTX 4070 SUPER
GPU / 12GB
PROMPT
13221
tokens/s
GENERATION
171
tokens/s
TTFT
105
ms
LOCALSCORE
2761
NVIDIA GeForce RTX 3080
GPU / 10GB
PROMPT
11541
tokens/s
GENERATION
206
tokens/s
TTFT
118
ms
LOCALSCORE
2711
Tesla V100-SXM2-32GB
GPU / 32GB
PROMPT
9892
tokens/s
GENERATION
249
tokens/s
TTFT
129
ms
LOCALSCORE
2672
Tesla V100-SXM2-16GB
GPU / 16GB
PROMPT
9743
tokens/s
GENERATION
250
tokens/s
TTFT
131
ms
LOCALSCORE
2647
NVIDIA GeForce RTX 4070 Ti
GPU / 12GB
PROMPT
12818
tokens/s
GENERATION
165
tokens/s
TTFT
122
ms
LOCALSCORE
2579
Quadro RTX 8000
GPU / 47GB
PROMPT
9356
tokens/s
GENERATION
250
tokens/s
TTFT
141
ms
LOCALSCORE
2550
Tesla V100-PCIE-16GB
GPU / 16GB
PROMPT
8984
tokens/s
GENERATION
248
tokens/s
TTFT
135
ms
LOCALSCORE
2545
NVIDIA H200
GPU / 140GB
PROMPT
14696
tokens/s
GENERATION
98.5
tokens/s
TTFT
89
ms
LOCALSCORE
2539
NVIDIA GeForce RTX 5060 Ti
GPU / 15GB
PROMPT
9969
tokens/s
GENERATION
227
tokens/s
TTFT
146
ms
LOCALSCORE
2497
NVIDIA H100 80GB HBM3
GPU / 79GB
PROMPT
14608
tokens/s
GENERATION
96.2
tokens/s
TTFT
91
ms
LOCALSCORE
2490
NVIDIA RTX 4000 Ada Generation
GPU / 20GB
PROMPT
10448
tokens/s
GENERATION
210
tokens/s
TTFT
144
ms
LOCALSCORE
2477
NVIDIA GeForce RTX 5080
GPU / 15GB
PROMPT
13606
tokens/s
GENERATION
103
tokens/s
TTFT
96
ms
LOCALSCORE
2447
NVIDIA GeForce RTX 5070 Ti
GPU / 15GB
PROMPT
15656
tokens/s
GENERATION
213
tokens/s
TTFT
603
ms
LOCALSCORE
2440
NVIDIA GeForce RTX 3070
GPU / 8GB
PROMPT
8503
tokens/s
GENERATION
230
tokens/s
TTFT
165
ms
LOCALSCORE
2277
NVIDIA GeForce RTX 2080 Ti
GPU / 11GB
PROMPT
8626
tokens/s
GENERATION
210
tokens/s
TTFT
153
ms
LOCALSCORE
2276
NVIDIA GeForce RTX 4060 Ti
GPU / 16GB
PROMPT
9203
tokens/s
GENERATION
209
tokens/s
TTFT
168
ms
LOCALSCORE
2254
NVIDIA TITAN V
GPU / 12GB
PROMPT
7853
tokens/s
GENERATION
232
tokens/s
TTFT
159
ms
LOCALSCORE
2252
NVIDIA H100 NVL
GPU / 93GB
PROMPT
13164
tokens/s
GENERATION
83.9
tokens/s
TTFT
99
ms
LOCALSCORE
2232
NVIDIA RTX A4000
GPU / 16GB
PROMPT
7961
tokens/s
GENERATION
229
tokens/s
TTFT
178
ms
LOCALSCORE
2175
NVIDIA L4
GPU / 22GB
PROMPT
9268
tokens/s
GENERATION
188
tokens/s
TTFT
171
ms
LOCALSCORE
2169
NVIDIA RTX 3500 Ada Generation Laptop GPU
GPU / 12GB
PROMPT
8206
tokens/s
GENERATION
181
tokens/s
TTFT
161
ms
LOCALSCORE
2099
NVIDIA RTX 4000 SFF Ada Generation
GPU / 20GB
PROMPT
8737
tokens/s
GENERATION
189
tokens/s
TTFT
179
ms
LOCALSCORE
2099
NVIDIA GeForce RTX 3060 Ti
GPU / 8GB
PROMPT
7588
tokens/s
GENERATION
184
tokens/s
TTFT
181
ms
LOCALSCORE
1972
NVIDIA A100 80GB PCIe
GPU / 79GB
PROMPT
10806
tokens/s
GENERATION
81.9
tokens/s
TTFT
117
ms
LOCALSCORE
1964
NVIDIA RTX A5000 Laptop GPU
GPU / 16GB
PROMPT
7018
tokens/s
GENERATION
202
tokens/s
TTFT
207
ms
LOCALSCORE
1900
NVIDIA GeForce RTX 2080
GPU / 8GB
PROMPT
6899
tokens/s
GENERATION
188
tokens/s
TTFT
195
ms
LOCALSCORE
1880
NVIDIA GeForce RTX 3080 Laptop GPU
GPU / 16GB
PROMPT
7533
tokens/s
GENERATION
157
tokens/s
TTFT
183
ms
LOCALSCORE
1865
NVIDIA GeForce RTX 2070 SUPER
GPU / 8GB
PROMPT
6350
tokens/s
GENERATION
212
tokens/s
TTFT
216
ms
LOCALSCORE
1840
NVIDIA A10
GPU / 22GB
PROMPT
8777
tokens/s
GENERATION
93.5
tokens/s
TTFT
153
ms
LOCALSCORE
1750
NVIDIA GeForce RTX 3060
GPU / 12GB
PROMPT
6062
tokens/s
GENERATION
200
tokens/s
TTFT
231
ms
LOCALSCORE
1737
NVIDIA GeForce RTX 2070
GPU / 8GB
PROMPT
5665
tokens/s
GENERATION
196
tokens/s
TTFT
234
ms
LOCALSCORE
1680
Apple M3 Ultra 24P+8E+80GPU
GPU / 512GB
PROMPT
5483
tokens/s
GENERATION
179
tokens/s
TTFT
209
ms
LOCALSCORE
1674
NVIDIA RTX 2000 Ada Generation
GPU / 16GB
PROMPT
6087
tokens/s
GENERATION
168
tokens/s
TTFT
251
ms
LOCALSCORE
1599
Apple M3 Ultra 24P+8E+80GPU
GPU / 256GB
PROMPT
4999
tokens/s
GENERATION
178
tokens/s
TTFT
227
ms
LOCALSCORE
1578
NVIDIA GeForce RTX 2060 SUPER
GPU / 8GB
PROMPT
5553
tokens/s
GENERATION
195
tokens/s
TTFT
287
ms
LOCALSCORE
1558
NVIDIA GeForce RTX 5090
GPU / 32GB
PROMPT
12376
tokens/s
GENERATION
60.3
tokens/s
TTFT
353
ms
LOCALSCORE
1508
NVIDIA RTX PRO 2000 Blackwell Generation Laptop GPU
GPU / 8GB
PROMPT
5884
tokens/s
GENERATION
131
tokens/s
TTFT
237
ms
LOCALSCORE
1481
NVIDIA GeForce RTX 4070 Laptop GPU
GPU / 8GB
PROMPT
6763
tokens/s
GENERATION
90.1
tokens/s
TTFT
213
ms
LOCALSCORE
1419
NVIDIA GeForce RTX 5080
GPU / 16GB
PROMPT
9480
tokens/s
GENERATION
41.2
tokens/s
TTFT
144
ms
LOCALSCORE
1394
NVIDIA GeForce RTX 4060 Laptop GPU
GPU / 8GB
PROMPT
6225
tokens/s
GENERATION
96.4
tokens/s
TTFT
224
ms
LOCALSCORE
1388
NVIDIA GeForce RTX 4060 Ti
GPU / 8GB
PROMPT
7267
tokens/s
GENERATION
70.5
tokens/s
TTFT
191
ms
LOCALSCORE
1388
NVIDIA GeForce RTX 3060 Laptop GPU
GPU / 6GB
PROMPT
4985
tokens/s
GENERATION
139
tokens/s
TTFT
278
ms
LOCALSCORE
1356
NVIDIA GeForce RTX 4060
GPU / 8GB
PROMPT
5352
tokens/s
GENERATION
134
tokens/s
TTFT
380
ms
LOCALSCORE
1331
NVIDIA GeForce RTX 2060
GPU / 6GB
PROMPT
4711
tokens/s
GENERATION
140
tokens/s
TTFT
286
ms
LOCALSCORE
1320
Apple M4 Max 12P+4E+40GPU
GPU / 64GB
PROMPT
3868
tokens/s
GENERATION
180
tokens/s
TTFT
303
ms
LOCALSCORE
1320
Apple M4 Max 12P+4E+40GPU
GPU / 128GB
PROMPT
3829
tokens/s
GENERATION
183
tokens/s
TTFT
305
ms
LOCALSCORE
1319
NVIDIA TITAN Xp
GPU / 12GB
PROMPT
4298
tokens/s
GENERATION
160
tokens/s
TTFT
301
ms
LOCALSCORE
1317
NVIDIA GeForce RTX 5070
GPU / 12GB
PROMPT
8328
tokens/s
GENERATION
101
tokens/s
TTFT
371
ms
LOCALSCORE
1312
Apple M4 Max 12P+4E+40GPU
GPU / 48GB
PROMPT
3751
tokens/s
GENERATION
179
tokens/s
TTFT
309
ms
LOCALSCORE
1295
NVIDIA GeForce RTX 4050 Laptop GPU
GPU / 6GB
PROMPT
4979
tokens/s
GENERATION
114
tokens/s
TTFT
286
ms
LOCALSCORE
1255
NVIDIA RTX A2000 12GB
GPU / 12GB
PROMPT
4363
tokens/s
GENERATION
144
tokens/s
TTFT
330
ms
LOCALSCORE
1241
NVIDIA GeForce GTX 1080 Ti
GPU / 11GB
PROMPT
4063
tokens/s
GENERATION
145
tokens/s
TTFT
318
ms
LOCALSCORE
1228
Apple M2 Ultra 16P+8E+60GPU
GPU / 128GB
PROMPT
3296
tokens/s
GENERATION
176
tokens/s
TTFT
334
ms
LOCALSCORE
1203
Apple M2 Ultra 16P+8E+60GPU
GPU / 192GB
PROMPT
3272
tokens/s
GENERATION
170
tokens/s
TTFT
339
ms
LOCALSCORE
1179
NVIDIA GeForce RTX 3050 Ti Laptop GPU
GPU / 4GB
PROMPT
4087
tokens/s
GENERATION
130
tokens/s
TTFT
354
ms
LOCALSCORE
1144
Apple M4 Max 10P+4E+32GPU
GPU / 36GB
PROMPT
3211
tokens/s
GENERATION
169
tokens/s
TTFT
371
ms
LOCALSCORE
1136
Apple M3 Max 12P+4E+40GPU
GPU / 48GB
PROMPT
3395
tokens/s
GENERATION
150
tokens/s
TTFT
348
ms
LOCALSCORE
1134
AMD Radeon PRO V620
GPU / 30GB
PROMPT
4608
tokens/s
GENERATION
102
tokens/s
TTFT
326
ms
LOCALSCORE
1129
Apple M3 Max 12P+4E+40GPU
GPU / 128GB
PROMPT
3292
tokens/s
GENERATION
146
tokens/s
TTFT
358
ms
LOCALSCORE
1104
NVIDIA RTX A2000
GPU / 6GB
PROMPT
3811
tokens/s
GENERATION
117
tokens/s
TTFT
349
ms
LOCALSCORE
1084
Apple M1 Ultra 16P+4E+64GPU
GPU / 128GB
PROMPT
2977
tokens/s
GENERATION
151
tokens/s
TTFT
368
ms
LOCALSCORE
1070
AMD Radeon RX 7900 XTX
GPU / 24GB
PROMPT
4241
tokens/s
GENERATION
106
tokens/s
TTFT
326
ms
LOCALSCORE
1069
NVIDIA RTX A1000 Laptop GPU
GPU / 4GB
PROMPT
3665
tokens/s
GENERATION
123
tokens/s
TTFT
401
ms
LOCALSCORE
1040
Radeon RX 7900 XT
GPU / 20GB
PROMPT
6502
tokens/s
GENERATION
34.6
tokens/s
TTFT
216
ms
LOCALSCORE
1013
AMD Radeon RX 9070 XT
GPU / 16GB
PROMPT
2708
tokens/s
GENERATION
163
tokens/s
TTFT
461
ms
LOCALSCORE
984
NVIDIA RTX A2000 8GB Laptop GPU
GPU / 8GB
PROMPT
3399
tokens/s
GENERATION
109
tokens/s
TTFT
416
ms
LOCALSCORE
962
Apple M2 Max 8P+4E+38GPU
GPU / 32GB
PROMPT
2551
tokens/s
GENERATION
153
tokens/s
TTFT
442
ms
LOCALSCORE
959
NVIDIA RTX A4500 Laptop GPU
GPU / 15GB
PROMPT
3868
tokens/s
GENERATION
72.5
tokens/s
TTFT
325
ms
LOCALSCORE
952
Apple M1 Ultra 16P+4E+48GPU
GPU / 128GB
PROMPT
2582
tokens/s
GENERATION
138
tokens/s
TTFT
425
ms
LOCALSCORE
943
NVIDIA GeForce GTX 1070 Ti
GPU / 8GB
PROMPT
2980
tokens/s
GENERATION
116
tokens/s
TTFT
441
ms
LOCALSCORE
922
NVIDIA GeForce RTX 3050
GPU / 6GB
PROMPT
3355
tokens/s
GENERATION
98.0
tokens/s
TTFT
423
ms
LOCALSCORE
920
NVIDIA RTX A2000 Laptop GPU
GPU / 4GB
PROMPT
3084
tokens/s
GENERATION
105
tokens/s
TTFT
431
ms
LOCALSCORE
910
Apple M3 Max 10P+4E+30GPU
GPU / 36GB
PROMPT
2553
tokens/s
GENERATION
133
tokens/s
TTFT
467
ms
LOCALSCORE
899
Radeon RX 7900 XTX
GPU / 24GB
PROMPT
4042
tokens/s
GENERATION
74.7
tokens/s
TTFT
354
ms
LOCALSCORE
898
NVIDIA RTX 500 Ada Generation Laptop GPU
GPU / 4GB
PROMPT
3624
tokens/s
GENERATION
81.6
tokens/s
TTFT
416
ms
LOCALSCORE
893
Apple M3 Max 10P+4E+30GPU
GPU / 96GB
PROMPT
2520
tokens/s
GENERATION
133
tokens/s
TTFT
476
ms
LOCALSCORE
889
Tesla P100-SXM2-16GB
GPU / 16GB
PROMPT
2516
tokens/s
GENERATION
124
tokens/s
TTFT
477
ms
LOCALSCORE
868
AMD Radeon RX 7800 XT
GPU / 16GB
PROMPT
2512
tokens/s
GENERATION
121
tokens/s
TTFT
490
ms
LOCALSCORE
852
AMD Radeon RX 6900 XT
GPU / 16GB
PROMPT
4411
tokens/s
GENERATION
62.8
tokens/s
TTFT
409
ms
LOCALSCORE
849
Apple M3 Max 12P+4E+40GPU
GPU / 64GB
PROMPT
2522
tokens/s
GENERATION
107
tokens/s
TTFT
521
ms
LOCALSCORE
803
AMD Radeon RX 6800 XT
GPU / 16GB
PROMPT
3955
tokens/s
GENERATION
60.9
tokens/s
TTFT
444
ms
LOCALSCORE
783
Apple M2 Max 8P+4E+30GPU
GPU / 32GB
PROMPT
2038
tokens/s
GENERATION
129
tokens/s
TTFT
554
ms
LOCALSCORE
781
Apple M1 Max 8P+2E+32GPU
GPU / 64GB
PROMPT
2060
tokens/s
GENERATION
124
tokens/s
TTFT
552
ms
LOCALSCORE
774
Apple M1 Max 8P+2E+32GPU
GPU / 32GB
PROMPT
2022
tokens/s
GENERATION
126
tokens/s
TTFT
560
ms
LOCALSCORE
768
Apple M4 Pro 10P+4E+20GPU
GPU / 24GB
PROMPT
2129
tokens/s
GENERATION
119
tokens/s
TTFT
573
ms
LOCALSCORE
763
Apple M4 Pro 10P+4E+20GPU
GPU / 48GB
PROMPT
2123
tokens/s
GENERATION
119
tokens/s
TTFT
576
ms
LOCALSCORE
760
AMD Radeon RX 6800
GPU / 16GB
PROMPT
3322
tokens/s
GENERATION
69.0
tokens/s
TTFT
486
ms
LOCALSCORE
759
NVIDIA GeForce GTX 1070
GPU / 8GB
PROMPT
2378
tokens/s
GENERATION
97.0
tokens/s
TTFT
543
ms
LOCALSCORE
752
Apple M4 Pro 10P+4E+20GPU
GPU / 64GB
PROMPT
2102
tokens/s
GENERATION
116
tokens/s
TTFT
582
ms
LOCALSCORE
749
AMD Radeon RX 6650 XT
GPU / 8GB
PROMPT
2633
tokens/s
GENERATION
87.4
tokens/s
TTFT
591
ms
LOCALSCORE
734
AMD Radeon RX 6700 XT
GPU / 12GB
PROMPT
2971
tokens/s
GENERATION
77.2
tokens/s
TTFT
537
ms
LOCALSCORE
729
Apple M4 Pro 8P+4E+16GPU
GPU / 24GB
PROMPT
1791
tokens/s
GENERATION
111
tokens/s
TTFT
676
ms
LOCALSCORE
664
Apple M4 Pro 8P+4E+16GPU
GPU / 48GB
PROMPT
1735
tokens/s
GENERATION
111
tokens/s
TTFT
683
ms
LOCALSCORE
655
Apple M1 Max 8P+2E+24GPU
GPU / 64GB
PROMPT
1682
tokens/s
GENERATION
106
tokens/s
TTFT
685
ms
LOCALSCORE
638
NVIDIA GeForce GTX 1060 6GB
GPU / 6GB
PROMPT
1863
tokens/s
GENERATION
76.9
tokens/s
TTFT
722
ms
LOCALSCORE
583
AMD Radeon RX 6600
GPU / 8GB
PROMPT
2052
tokens/s
GENERATION
72.9
tokens/s
TTFT
768
ms
LOCALSCORE
580
Apple M2 Pro 8P+4E+19GPU
GPU / 32GB
PROMPT
1491
tokens/s
GENERATION
100
tokens/s
TTFT
806
ms
LOCALSCORE
570
Apple M3 Pro 6P+6E+18GPU
GPU / 36GB
PROMPT
1586
tokens/s
GENERATION
89.8
tokens/s
TTFT
809
ms
LOCALSCORE
560
Apple M2 Pro 6P+4E+16GPU
GPU / 16GB
PROMPT
1328
tokens/s
GENERATION
91.1
tokens/s
TTFT
902
ms
LOCALSCORE
512
Apple M2 Pro 6P+4E+16GPU
GPU / 32GB
PROMPT
1281
tokens/s
GENERATION
91.5
tokens/s
TTFT
914
ms
LOCALSCORE
504
Apple M3 Pro 5P+6E+14GPU
GPU / 18GB
PROMPT
1344
tokens/s
GENERATION
88.1
tokens/s
TTFT
959
ms
LOCALSCORE
498
Apple M3 Pro 5P+6E+14GPU
GPU / 36GB
PROMPT
1328
tokens/s
GENERATION
88.2
tokens/s
TTFT
953
ms
LOCALSCORE
497
Apple M5 4P+6E+10GPU
GPU / 16GB
PROMPT
1217
tokens/s
GENERATION
97.1
tokens/s
TTFT
1.01
sec
LOCALSCORE
489
AMD Radeon RX 7600 XT
GPU / 8GB
PROMPT
1502
tokens/s
GENERATION
57.1
tokens/s
TTFT
903
ms
LOCALSCORE
456
Apple M1 Pro 8P+2E+16GPU
GPU / 16GB
PROMPT
1168
tokens/s
GENERATION
78.9
tokens/s
TTFT
1.02
sec
LOCALSCORE
448
NVIDIA GeForce RTX 2050
GPU / 4GB
PROMPT
1770
tokens/s
GENERATION
47.9
tokens/s
TTFT
967
ms
LOCALSCORE
444
Apple M1 Pro 8P+2E+16GPU
GPU / 32GB
PROMPT
1166
tokens/s
GENERATION
77.2
tokens/s
TTFT
1.04
sec
LOCALSCORE
443
Quadro P2000
GPU / 5GB
PROMPT
1353
tokens/s
GENERATION
58.3
tokens/s
TTFT
1.01
sec
LOCALSCORE
428
Apple M4 4P+6E+10GPU
GPU / 16GB
PROMPT
1100
tokens/s
GENERATION
76.5
tokens/s
TTFT
1.19
sec
LOCALSCORE
414
Apple M1 Pro 6P+2E+14GPU
GPU / 32GB
PROMPT
1064
tokens/s
GENERATION
71.0
tokens/s
TTFT
1.13
sec
LOCALSCORE
405
Apple M1 Pro 6P+2E+14GPU
GPU / 16GB
PROMPT
1033
tokens/s
GENERATION
71.6
tokens/s
TTFT
1.16
sec
LOCALSCORE
400
Apple M4 4P+6E+10GPU
GPU / 32GB
PROMPT
1028
tokens/s
GENERATION
74.6
tokens/s
TTFT
1.25
sec
LOCALSCORE
395
Apple M4 4P+6E+10GPU
GPU / 24GB
PROMPT
982
tokens/s
GENERATION
74.4
tokens/s
TTFT
1.30
sec
LOCALSCORE
383
Orin
GPU / 7GB
PROMPT
1233
tokens/s
GENERATION
47.8
tokens/s
TTFT
1.18
sec
LOCALSCORE
369
Apple M3 4P+4E+10GPU
GPU / 16GB
PROMPT
927
tokens/s
GENERATION
67.1
tokens/s
TTFT
1.42
sec
LOCALSCORE
353
Apple M3 4P+4E+10GPU
GPU / 24GB
PROMPT
917
tokens/s
GENERATION
64.7
tokens/s
TTFT
1.43
sec
LOCALSCORE
346
Apple M4 4P+6E+8GPU
GPU / 16GB
PROMPT
894
tokens/s
GENERATION
65.6
tokens/s
TTFT
1.45
sec
LOCALSCORE
344
NVIDIA GeForce GTX 1080
GPU / 8GB
PROMPT
1057
tokens/s
GENERATION
46.9
tokens/s
TTFT
1.22
sec
LOCALSCORE
343
NVIDIA GeForce GTX 1050 Ti
GPU / 4GB
PROMPT
1013
tokens/s
GENERATION
44.5
tokens/s
TTFT
1.33
sec
LOCALSCORE
324
Apple M2 4P+4E+10GPU
GPU / 24GB
PROMPT
827
tokens/s
GENERATION
56.9
tokens/s
TTFT
1.57
sec
LOCALSCORE
310
Apple M2 4P+4E+10GPU
GPU / 8GB
PROMPT
805
tokens/s
GENERATION
56.5
tokens/s
TTFT
1.67
sec
LOCALSCORE
301
Apple M2 4P+4E+10GPU
GPU / 16GB
PROMPT
780
tokens/s
GENERATION
51.6
tokens/s
TTFT
1.60
sec
LOCALSCORE
293
AMD Radeon Graphics
GPU / 32GB
PROMPT
599
tokens/s
GENERATION
72.6
tokens/s
TTFT
2.21
sec
LOCALSCORE
270
NVIDIA GeForce GTX 1660 SUPER
GPU / 6GB
PROMPT
825
tokens/s
GENERATION
36.5
tokens/s
TTFT
1.75
sec
LOCALSCORE
258
NVIDIA GeForce GTX 1050
GPU / 4GB
PROMPT
742
tokens/s
GENERATION
31.1
tokens/s
TTFT
1.71
sec
LOCALSCORE
239
Apple M1 4P+4E+8GPU
GPU / 8GB
PROMPT
606
tokens/s
GENERATION
39.8
tokens/s
TTFT
2.17
sec
LOCALSCORE
223
NVIDIA GeForce GTX 1070 with Max-Q Design
GPU / 8GB
PROMPT
814
tokens/s
GENERATION
24.4
tokens/s
TTFT
1.83
sec
LOCALSCORE
221
Apple M1 4P+4E+8GPU
GPU / 16GB
PROMPT
586
tokens/s
GENERATION
40.2
tokens/s
TTFT
2.25
sec
LOCALSCORE
219
Apple M1 4P+4E+7GPU
GPU / 16GB
PROMPT
538
tokens/s
GENERATION
38.5
tokens/s
TTFT
2.41
sec
LOCALSCORE
205
Apple M1 4P+4E+7GPU
GPU / 8GB
PROMPT
533
tokens/s
GENERATION
38.5
tokens/s
TTFT
2.43
sec
LOCALSCORE
204
Apple M2 4P+4E+8GPU
GPU / 8GB
PROMPT
523
tokens/s
GENERATION
34.5
tokens/s
TTFT
2.72
sec
LOCALSCORE
194
NVIDIA GeForce GTX 1050
GPU / 2GB
PROMPT
539
tokens/s
GENERATION
27.4
tokens/s
TTFT
6.24
sec
LOCALSCORE
178
NVIDIA GeForce GT 1030
GPU / 2GB
PROMPT
543
tokens/s
GENERATION
24.0
tokens/s
TTFT
2.60
sec
LOCALSCORE
171
NVIDIA GeForce GTX 1650 Ti
GPU / 4GB
PROMPT
567
tokens/s
GENERATION
20.7
tokens/s
TTFT
2.58
sec
LOCALSCORE
166
NVIDIA T1200 Laptop GPU
GPU / 4GB
PROMPT
540
tokens/s
GENERATION
21.0
tokens/s
TTFT
2.61
sec
LOCALSCORE
163
AMD Radeon 780M
GPU / 27GB
PROMPT
690
tokens/s
GENERATION
11.7
tokens/s
TTFT
1.88
sec
LOCALSCORE
162
AMD Radeon 780M Graphics
GPU / 14GB
PROMPT
666
tokens/s
GENERATION
11.8
tokens/s
TTFT
1.96
sec
LOCALSCORE
159
AMD Radeon(TM) 780M
GPU / 29GB
PROMPT
617
tokens/s
GENERATION
11.5
tokens/s
TTFT
2.09
sec
LOCALSCORE
150
AMD Radeon(TM) Graphics
GPU / 25GB
PROMPT
565
tokens/s
GENERATION
10.7
tokens/s
TTFT
2.29
sec
LOCALSCORE
138
NVIDIA GeForce MX150
GPU / 2GB
PROMPT
311
tokens/s
GENERATION
13.5
tokens/s
TTFT
4.06
sec
LOCALSCORE
101
Radeon RX 5500 XT
GPU / 4GB
PROMPT
174
tokens/s
GENERATION
56.2
tokens/s
TTFT
11.45
sec
LOCALSCORE
95
AMD Radeon 780M Graphics
GPU / 19GB
PROMPT
399
tokens/s
GENERATION
5.9
tokens/s
TTFT
3.36
sec
LOCALSCORE
89