logo
HomeLatest ResultsDownloadAboutBlog
model icon

Llama 3.2 1B Instruct

Q4_K - Medium

1.5Bparams
COMPARE ACCELERATORS

717 accelerators tested

Select Accelerators

NVIDIA RTX 6000 Ada Generation

47GB

NVIDIA GeForce RTX 4090

23GB

NVIDIA GeForce RTX 4090 D

47GB

NVIDIA L40S

44GB

NVIDIA RTX PRO 6000 Blackwell Workstation Edition

95GB

Llama 3.2 1B Instruct - Q4_K - Medium

LEADERBOARD
NVIDIA RTX 6000 Ada Generation
GPU / 47GB
PROMPT
27281
tokens/s
GENERATION
411
tokens/s
TTFT
53
ms
LOCALSCORE
5948
NVIDIA GeForce RTX 4090
GPU / 23GB
PROMPT
25538
tokens/s
GENERATION
382
tokens/s
TTFT
57
ms
LOCALSCORE
5562
NVIDIA GeForce RTX 4090 D
GPU / 47GB
PROMPT
24549
tokens/s
GENERATION
359
tokens/s
TTFT
57
ms
LOCALSCORE
5380
NVIDIA L40S
GPU / 44GB
PROMPT
24809
tokens/s
GENERATION
355
tokens/s
TTFT
61
ms
LOCALSCORE
5252
NVIDIA RTX PRO 6000 Blackwell Workstation Edition
GPU / 95GB
PROMPT
25401
tokens/s
GENERATION
244
tokens/s
TTFT
55
ms
LOCALSCORE
4842
NVIDIA H100 PCIe
GPU / 79GB
PROMPT
18590
tokens/s
GENERATION
335
tokens/s
TTFT
68
ms
LOCALSCORE
4512
NVIDIA GeForce RTX 3090 Ti
GPU / 24GB
PROMPT
15120
tokens/s
GENERATION
354
tokens/s
TTFT
90
ms
LOCALSCORE
3908
NVIDIA A100-SXM4-80GB
GPU / 79GB
PROMPT
15370
tokens/s
GENERATION
308
tokens/s
TTFT
80
ms
LOCALSCORE
3892
NVIDIA GeForce RTX 4080 SUPER
GPU / 32GB
PROMPT
17479
tokens/s
GENERATION
247
tokens/s
TTFT
80
ms
LOCALSCORE
3780
NVIDIA GeForce RTX 4090
GPU / 24GB
PROMPT
18869
tokens/s
GENERATION
194
tokens/s
TTFT
102
ms
LOCALSCORE
3523
NVIDIA GeForce RTX 3080 Ti
GPU / 12GB
PROMPT
13824
tokens/s
GENERATION
288
tokens/s
TTFT
97
ms
LOCALSCORE
3443
NVIDIA RTX A6000
GPU / 48GB
PROMPT
13191
tokens/s
GENERATION
315
tokens/s
TTFT
102
ms
LOCALSCORE
3438
NVIDIA RTX 6000 Ada Generation
GPU / 48GB
PROMPT
19620
tokens/s
GENERATION
131
tokens/s
TTFT
68
ms
LOCALSCORE
3350
NVIDIA GeForce RTX 5090
GPU / 31GB
PROMPT
20305
tokens/s
GENERATION
170
tokens/s
TTFT
300
ms
LOCALSCORE
3245
NVIDIA RTX A5000
GPU / 24GB
PROMPT
12138
tokens/s
GENERATION
310
tokens/s
TTFT
110
ms
LOCALSCORE
3242
NVIDIA GeForce RTX 3090
GPU / 24GB
PROMPT
13267
tokens/s
GENERATION
258
tokens/s
TTFT
102
ms
LOCALSCORE
3195
NVIDIA GeForce RTX 4070
GPU / 12GB
PROMPT
12717
tokens/s
GENERATION
283
tokens/s
TTFT
112
ms
LOCALSCORE
3179
NVIDIA GeForce RTX 4080
GPU / 16GB
PROMPT
14496
tokens/s
GENERATION
241
tokens/s
TTFT
130
ms
LOCALSCORE
3133
NVIDIA A40
GPU / 44GB
PROMPT
12156
tokens/s
GENERATION
284
tokens/s
TTFT
113
ms
LOCALSCORE
3126
NVIDIA L40S
GPU / 45GB
PROMPT
18255
tokens/s
GENERATION
121
tokens/s
TTFT
75
ms
LOCALSCORE
3088
NVIDIA GeForce RTX 4080 SUPER
GPU / 16GB
PROMPT
16736
tokens/s
GENERATION
141
tokens/s
TTFT
81
ms
LOCALSCORE
3077
NVIDIA L40
GPU / 44GB
PROMPT
17686
tokens/s
GENERATION
118
tokens/s
TTFT
77
ms
LOCALSCORE
3000
NVIDIA GeForce RTX 3080
GPU / 20GB
PROMPT
11749
tokens/s
GENERATION
255
tokens/s
TTFT
113
ms
LOCALSCORE
2974
NVIDIA RTX PRO 6000 Blackwell Server Edition
GPU / 95GB
PROMPT
18081
tokens/s
GENERATION
106
tokens/s
TTFT
74
ms
LOCALSCORE
2962
NVIDIA RTX A4500
GPU / 20GB
PROMPT
10422
tokens/s
GENERATION
286
tokens/s
TTFT
131
ms
LOCALSCORE
2832
NVIDIA GeForce RTX 4070 SUPER
GPU / 12GB
PROMPT
13280
tokens/s
GENERATION
183
tokens/s
TTFT
106
ms
LOCALSCORE
2821
NVIDIA GeForce RTX 3070 Ti
GPU / 8GB
PROMPT
10074
tokens/s
GENERATION
297
tokens/s
TTFT
137
ms
LOCALSCORE
2795
NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
GPU / 95GB
PROMPT
16184
tokens/s
GENERATION
106
tokens/s
TTFT
80
ms
LOCALSCORE
2780
NVIDIA GeForce RTX 4070 Ti SUPER
GPU / 16GB
PROMPT
13268
tokens/s
GENERATION
161
tokens/s
TTFT
104
ms
LOCALSCORE
2738
Tesla V100-SXM2-32GB
GPU / 32GB
PROMPT
9908
tokens/s
GENERATION
261
tokens/s
TTFT
127
ms
LOCALSCORE
2726
Tesla V100-SXM2-16GB
GPU / 16GB
PROMPT
9801
tokens/s
GENERATION
264
tokens/s
TTFT
129
ms
LOCALSCORE
2719
NVIDIA GeForce RTX 3080
GPU / 10GB
PROMPT
11436
tokens/s
GENERATION
208
tokens/s
TTFT
119
ms
LOCALSCORE
2703
NVIDIA GeForce RTX 5070 Ti Laptop GPU
GPU / 12GB
PROMPT
11428
tokens/s
GENERATION
186
tokens/s
TTFT
119
ms
LOCALSCORE
2612
NVIDIA GeForce RTX 4070 Ti
GPU / 12GB
PROMPT
12818
tokens/s
GENERATION
165
tokens/s
TTFT
122
ms
LOCALSCORE
2579
NVIDIA GeForce RTX 5090 Laptop GPU
GPU / 24GB
PROMPT
11598
tokens/s
GENERATION
182
tokens/s
TTFT
125
ms
LOCALSCORE
2565
NVIDIA GeForce RTX 4090 Laptop GPU
GPU / 16GB
PROMPT
11965
tokens/s
GENERATION
207
tokens/s
TTFT
148
ms
LOCALSCORE
2559
NVIDIA GeForce RTX 5070 Ti
GPU / 15GB
PROMPT
14657
tokens/s
GENERATION
185
tokens/s
TTFT
380
ms
LOCALSCORE
2559
Quadro RTX 8000
GPU / 47GB
PROMPT
9356
tokens/s
GENERATION
250
tokens/s
TTFT
141
ms
LOCALSCORE
2550
Tesla V100-PCIE-16GB
GPU / 16GB
PROMPT
8984
tokens/s
GENERATION
248
tokens/s
TTFT
135
ms
LOCALSCORE
2545
NVIDIA H200
GPU / 140GB
PROMPT
14696
tokens/s
GENERATION
98.5
tokens/s
TTFT
89
ms
LOCALSCORE
2539
Quadro RTX 6000
GPU / 23GB
PROMPT
9138
tokens/s
GENERATION
253
tokens/s
TTFT
143
ms
LOCALSCORE
2526
NVIDIA H100 80GB HBM3
GPU / 79GB
PROMPT
14608
tokens/s
GENERATION
96.2
tokens/s
TTFT
91
ms
LOCALSCORE
2490
NVIDIA RTX 4000 Ada Generation
GPU / 20GB
PROMPT
10448
tokens/s
GENERATION
210
tokens/s
TTFT
144
ms
LOCALSCORE
2477
NVIDIA GeForce RTX 5080
GPU / 15GB
PROMPT
13606
tokens/s
GENERATION
103
tokens/s
TTFT
96
ms
LOCALSCORE
2447
NVIDIA GeForce RTX 2080 Ti
GPU / 11GB
PROMPT
8722
tokens/s
GENERATION
222
tokens/s
TTFT
151
ms
LOCALSCORE
2337
NVIDIA H100 NVL
GPU / 93GB
PROMPT
13586
tokens/s
GENERATION
83.6
tokens/s
TTFT
97
ms
LOCALSCORE
2268
NVIDIA GeForce RTX 4060 Ti
GPU / 16GB
PROMPT
9203
tokens/s
GENERATION
209
tokens/s
TTFT
168
ms
LOCALSCORE
2254
NVIDIA TITAN V
GPU / 12GB
PROMPT
7853
tokens/s
GENERATION
232
tokens/s
TTFT
159
ms
LOCALSCORE
2252
Quadro RTX 8000
GPU / 44GB
PROMPT
8204
tokens/s
GENERATION
218
tokens/s
TTFT
163
ms
LOCALSCORE
2223
NVIDIA L4
GPU / 22GB
PROMPT
9268
tokens/s
GENERATION
188
tokens/s
TTFT
171
ms
LOCALSCORE
2169
NVIDIA GeForce RTX 3070
GPU / 8GB
PROMPT
8069
tokens/s
GENERATION
216
tokens/s
TTFT
181
ms
LOCALSCORE
2139
NVIDIA RTX 4000 SFF Ada Generation
GPU / 20GB
PROMPT
8753
tokens/s
GENERATION
189
tokens/s
TTFT
179
ms
LOCALSCORE
2100
NVIDIA RTX 3500 Ada Generation Laptop GPU
GPU / 12GB
PROMPT
8206
tokens/s
GENERATION
181
tokens/s
TTFT
161
ms
LOCALSCORE
2099
NVIDIA GeForce RTX 3060 Ti
GPU / 8GB
PROMPT
7644
tokens/s
GENERATION
193
tokens/s
TTFT
181
ms
LOCALSCORE
2009
NVIDIA RTX A4000
GPU / 16GB
PROMPT
7736
tokens/s
GENERATION
185
tokens/s
TTFT
181
ms
LOCALSCORE
1983
NVIDIA GeForce RTX 5060 Ti
GPU / 15GB
PROMPT
9504
tokens/s
GENERATION
210
tokens/s
TTFT
648
ms
LOCALSCORE
1978
NVIDIA A100 80GB PCIe
GPU / 79GB
PROMPT
10806
tokens/s
GENERATION
81.9
tokens/s
TTFT
117
ms
LOCALSCORE
1964
NVIDIA L40S-8Q
GPU / 8GB
PROMPT
12563
tokens/s
GENERATION
62.4
tokens/s
TTFT
110
ms
LOCALSCORE
1925
NVIDIA GeForce RTX 2080
GPU / 8GB
PROMPT
6821
tokens/s
GENERATION
205
tokens/s
TTFT
197
ms
LOCALSCORE
1920
NVIDIA GeForce RTX 2080 SUPER
GPU / 8GB
PROMPT
7096
tokens/s
GENERATION
186
tokens/s
TTFT
189
ms
LOCALSCORE
1911
NVIDIA GeForce RTX 5090
GPU / 32GB
PROMPT
13997
tokens/s
GENERATION
75.6
tokens/s
TTFT
294
ms
LOCALSCORE
1903
NVIDIA RTX A5000 Laptop GPU
GPU / 16GB
PROMPT
7018
tokens/s
GENERATION
202
tokens/s
TTFT
207
ms
LOCALSCORE
1900
NVIDIA GeForce RTX 5070 Ti
GPU / 16GB
PROMPT
9863
tokens/s
GENERATION
92.8
tokens/s
TTFT
139
ms
LOCALSCORE
1866
NVIDIA GeForce RTX 3080 Laptop GPU
GPU / 16GB
PROMPT
7533
tokens/s
GENERATION
157
tokens/s
TTFT
183
ms
LOCALSCORE
1865
NVIDIA GeForce RTX 2070 SUPER
GPU / 8GB
PROMPT
6350
tokens/s
GENERATION
212
tokens/s
TTFT
216
ms
LOCALSCORE
1840
NVIDIA A10
GPU / 22GB
PROMPT
8777
tokens/s
GENERATION
93.5
tokens/s
TTFT
153
ms
LOCALSCORE
1750
NVIDIA GeForce RTX 4070 Laptop GPU
GPU / 8GB
PROMPT
7398
tokens/s
GENERATION
143
tokens/s
TTFT
201
ms
LOCALSCORE
1723
NVIDIA GeForce RTX 4060 Ti
GPU / 8GB
PROMPT
7937
tokens/s
GENERATION
123
tokens/s
TTFT
184
ms
LOCALSCORE
1716
NVIDIA GeForce RTX 5060 Ti
GPU / 16GB
PROMPT
8432
tokens/s
GENERATION
100
tokens/s
TTFT
171
ms
LOCALSCORE
1704
Apple M3 Ultra 24P+8E+80GPU
GPU / 512GB
PROMPT
5601
tokens/s
GENERATION
179
tokens/s
TTFT
206
ms
LOCALSCORE
1693
NVIDIA GeForce RTX 2070
GPU / 8GB
PROMPT
5816
tokens/s
GENERATION
190
tokens/s
TTFT
230
ms
LOCALSCORE
1687
NVIDIA GB10
GPU / 122GB
PROMPT
7214
tokens/s
GENERATION
142
tokens/s
TTFT
216
ms
LOCALSCORE
1683
NVIDIA GeForce RTX 3060
GPU / 12GB
PROMPT
6064
tokens/s
GENERATION
184
tokens/s
TTFT
232
ms
LOCALSCORE
1678
NVIDIA GeForce RTX 2070 Super
GPU / 8GB
PROMPT
5696
tokens/s
GENERATION
189
tokens/s
TTFT
236
ms
LOCALSCORE
1660
NVIDIA RTX 2000 Ada Generation
GPU / 16GB
PROMPT
6087
tokens/s
GENERATION
168
tokens/s
TTFT
251
ms
LOCALSCORE
1599
Apple M3 Ultra 24P+8E+80GPU
GPU / 256GB
PROMPT
4999
tokens/s
GENERATION
178
tokens/s
TTFT
227
ms
LOCALSCORE
1578
NVIDIA GeForce RTX 2060 SUPER
GPU / 8GB
PROMPT
5553
tokens/s
GENERATION
195
tokens/s
TTFT
287
ms
LOCALSCORE
1558
NVIDIA GeForce RTX 4060 Laptop GPU
GPU / 8GB
PROMPT
6467
tokens/s
GENERATION
128
tokens/s
TTFT
219
ms
LOCALSCORE
1545
NVIDIA GB10
GPU / 120GB
PROMPT
6729
tokens/s
GENERATION
116
tokens/s
TTFT
235
ms
LOCALSCORE
1504
NVIDIA RTX PRO 2000 Blackwell Generation Laptop GPU
GPU / 8GB
PROMPT
5884
tokens/s
GENERATION
131
tokens/s
TTFT
237
ms
LOCALSCORE
1481
NVIDIA GeForce RTX 3060 Laptop GPU
GPU / 6GB
PROMPT
5430
tokens/s
GENERATION
151
tokens/s
TTFT
258
ms
LOCALSCORE
1475
NVIDIA GeForce RTX 5080
GPU / 16GB
PROMPT
9480
tokens/s
GENERATION
41.2
tokens/s
TTFT
144
ms
LOCALSCORE
1394
Apple M5 Max 6P+12E+32GPU
GPU / 36GB
PROMPT
3674
tokens/s
GENERATION
230
tokens/s
TTFT
320
ms
LOCALSCORE
1382
NVIDIA GeForce RTX 3070 Laptop GPU
GPU / 8GB
PROMPT
4930
tokens/s
GENERATION
124
tokens/s
TTFT
251
ms
LOCALSCORE
1347
NVIDIA GeForce RTX 4060
GPU / 8GB
PROMPT
5352
tokens/s
GENERATION
134
tokens/s
TTFT
380
ms
LOCALSCORE
1331
Apple M4 Max 12P+4E+40GPU
GPU / 128GB
PROMPT
3836
tokens/s
GENERATION
183
tokens/s
TTFT
304
ms
LOCALSCORE
1321
NVIDIA TITAN Xp
GPU / 12GB
PROMPT
4298
tokens/s
GENERATION
160
tokens/s
TTFT
301
ms
LOCALSCORE
1317
NVIDIA GeForce RTX 5070
GPU / 12GB
PROMPT
8328
tokens/s
GENERATION
101
tokens/s
TTFT
371
ms
LOCALSCORE
1312
Apple M4 Max 12P+4E+40GPU
GPU / 64GB
PROMPT
3817
tokens/s
GENERATION
180
tokens/s
TTFT
307
ms
LOCALSCORE
1308
Apple M4 Max 12P+4E+40GPU
GPU / 48GB
PROMPT
3751
tokens/s
GENERATION
179
tokens/s
TTFT
309
ms
LOCALSCORE
1295
NVIDIA RTX A2000 12GB
GPU / 12GB
PROMPT
4549
tokens/s
GENERATION
140
tokens/s
TTFT
314
ms
LOCALSCORE
1264
NVIDIA GeForce RTX 2060
GPU / 6GB
PROMPT
4479
tokens/s
GENERATION
137
tokens/s
TTFT
306
ms
LOCALSCORE
1263
NVIDIA GeForce RTX 4050 Laptop GPU
GPU / 6GB
PROMPT
4917
tokens/s
GENERATION
118
tokens/s
TTFT
301
ms
LOCALSCORE
1248
NVIDIA GeForce GTX 1080 Ti
GPU / 11GB
PROMPT
4098
tokens/s
GENERATION
147
tokens/s
TTFT
312
ms
LOCALSCORE
1245
Apple M2 Ultra 16P+8E+60GPU
GPU / 128GB
PROMPT
3296
tokens/s
GENERATION
176
tokens/s
TTFT
334
ms
LOCALSCORE
1203
Apple M2 Ultra 16P+8E+60GPU
GPU / 64GB
PROMPT
3290
tokens/s
GENERATION
174
tokens/s
TTFT
335
ms
LOCALSCORE
1196
Apple M2 Ultra 16P+8E+60GPU
GPU / 192GB
PROMPT
3272
tokens/s
GENERATION
170
tokens/s
TTFT
339
ms
LOCALSCORE
1179
NVIDIA GeForce RTX 3050 Ti Laptop GPU
GPU / 4GB
PROMPT
4087
tokens/s
GENERATION
130
tokens/s
TTFT
354
ms
LOCALSCORE
1144
Radeon RX 7900 XT
GPU / 20GB
PROMPT
4957
tokens/s
GENERATION
99.2
tokens/s
TTFT
280
ms
LOCALSCORE
1143
Apple M4 Max 10P+4E+32GPU
GPU / 36GB
PROMPT
3274
tokens/s
GENERATION
166
tokens/s
TTFT
368
ms
LOCALSCORE
1140
Apple M3 Max 12P+4E+40GPU
GPU / 48GB
PROMPT
3382
tokens/s
GENERATION
150
tokens/s
TTFT
352
ms
LOCALSCORE
1131
AMD Radeon PRO V620
GPU / 30GB
PROMPT
4608
tokens/s
GENERATION
102
tokens/s
TTFT
326
ms
LOCALSCORE
1129
Apple M3 Max 12P+4E+40GPU
GPU / 128GB
PROMPT
3292
tokens/s
GENERATION
146
tokens/s
TTFT
358
ms
LOCALSCORE
1104
NVIDIA RTX A2000
GPU / 6GB
PROMPT
3811
tokens/s
GENERATION
117
tokens/s
TTFT
349
ms
LOCALSCORE
1084
Apple M1 Ultra 16P+4E+64GPU
GPU / 128GB
PROMPT
2977
tokens/s
GENERATION
151
tokens/s
TTFT
368
ms
LOCALSCORE
1070
AMD Radeon RX 7900 XTX
GPU / 24GB
PROMPT
4241
tokens/s
GENERATION
106
tokens/s
TTFT
326
ms
LOCALSCORE
1069
NVIDIA RTX A1000 Laptop GPU
GPU / 4GB
PROMPT
3665
tokens/s
GENERATION
123
tokens/s
TTFT
401
ms
LOCALSCORE
1040
AMD Radeon RX 9070 XT
GPU / 16GB
PROMPT
2708
tokens/s
GENERATION
163
tokens/s
TTFT
461
ms
LOCALSCORE
984
AMD Radeon RX 6800 XT
GPU / 16GB
PROMPT
4282
tokens/s
GENERATION
84.0
tokens/s
TTFT
386
ms
LOCALSCORE
964
NVIDIA RTX A2000 8GB Laptop GPU
GPU / 8GB
PROMPT
3399
tokens/s
GENERATION
109
tokens/s
TTFT
416
ms
LOCALSCORE
962
NVIDIA RTX A4500 Laptop GPU
GPU / 15GB
PROMPT
3868
tokens/s
GENERATION
72.5
tokens/s
TTFT
325
ms
LOCALSCORE
952
NVIDIA GeForce RTX 5080 Laptop GPU
GPU / 16GB
PROMPT
5678
tokens/s
GENERATION
34.7
tokens/s
TTFT
231
ms
LOCALSCORE
949
Apple M5 Pro 6P+12E+20GPU
GPU / 24GB
PROMPT
2413
tokens/s
GENERATION
175
tokens/s
TTFT
500
ms
LOCALSCORE
946
Apple M2 Max 8P+4E+38GPU
GPU / 32GB
PROMPT
2523
tokens/s
GENERATION
150
tokens/s
TTFT
449
ms
LOCALSCORE
945
Apple M5 Pro 6P+12E+20GPU
GPU / 64GB
PROMPT
2410
tokens/s
GENERATION
174
tokens/s
TTFT
499
ms
LOCALSCORE
944
Apple M1 Ultra 16P+4E+48GPU
GPU / 128GB
PROMPT
2582
tokens/s
GENERATION
138
tokens/s
TTFT
425
ms
LOCALSCORE
943
NVIDIA GeForce RTX 3050
GPU / 6GB
PROMPT
3347
tokens/s
GENERATION
104
tokens/s
TTFT
423
ms
LOCALSCORE
936
NVIDIA GeForce GTX 1070 Ti
GPU / 8GB
PROMPT
2980
tokens/s
GENERATION
116
tokens/s
TTFT
441
ms
LOCALSCORE
922
NVIDIA GeForce RTX 3050 6GB Laptop GPU
GPU / 6GB
PROMPT
3376
tokens/s
GENERATION
99.4
tokens/s
TTFT
435
ms
LOCALSCORE
920
NVIDIA RTX A2000 Laptop GPU
GPU / 4GB
PROMPT
3084
tokens/s
GENERATION
105
tokens/s
TTFT
431
ms
LOCALSCORE
910
Apple M3 Max 10P+4E+30GPU
GPU / 36GB
PROMPT
2553
tokens/s
GENERATION
133
tokens/s
TTFT
467
ms
LOCALSCORE
899
Radeon RX 7900 XTX
GPU / 24GB
PROMPT
4042
tokens/s
GENERATION
74.7
tokens/s
TTFT
354
ms
LOCALSCORE
898
NVIDIA RTX 500 Ada Generation Laptop GPU
GPU / 4GB
PROMPT
3624
tokens/s
GENERATION
81.6
tokens/s
TTFT
416
ms
LOCALSCORE
893
Apple M3 Max 10P+4E+30GPU
GPU / 96GB
PROMPT
2520
tokens/s
GENERATION
133
tokens/s
TTFT
476
ms
LOCALSCORE
889
Tesla P100-SXM2-16GB
GPU / 16GB
PROMPT
2516
tokens/s
GENERATION
124
tokens/s
TTFT
477
ms
LOCALSCORE
868
AMD Radeon RX 7800 XT
GPU / 16GB
PROMPT
2512
tokens/s
GENERATION
121
tokens/s
TTFT
490
ms
LOCALSCORE
852
AMD Radeon RX 6900 XT
GPU / 16GB
PROMPT
4411
tokens/s
GENERATION
62.8
tokens/s
TTFT
409
ms
LOCALSCORE
849
Apple M3 Max 12P+4E+40GPU
GPU / 64GB
PROMPT
2522
tokens/s
GENERATION
107
tokens/s
TTFT
521
ms
LOCALSCORE
803
AMD Radeon RX 7800 XT
GPU / 16GB
PROMPT
2934
tokens/s
GENERATION
112
tokens/s
TTFT
767
ms
LOCALSCORE
789
Tesla P100-PCIE-16GB
GPU / 16GB
PROMPT
2244
tokens/s
GENERATION
115
tokens/s
TTFT
538
ms
LOCALSCORE
783
NVIDIA GeForce GTX 1070
GPU / 8GB
PROMPT
2476
tokens/s
GENERATION
100
tokens/s
TTFT
523
ms
LOCALSCORE
780
Apple M2 Max 8P+4E+30GPU
GPU / 32GB
PROMPT
2032
tokens/s
GENERATION
128
tokens/s
TTFT
558
ms
LOCALSCORE
775
Apple M1 Max 8P+2E+32GPU
GPU / 32GB
PROMPT
2023
tokens/s
GENERATION
126
tokens/s
TTFT
560
ms
LOCALSCORE
769
Apple M4 Pro 10P+4E+20GPU
GPU / 64GB
PROMPT
2145
tokens/s
GENERATION
119
tokens/s
TTFT
572
ms
LOCALSCORE
763
Apple M4 Pro 10P+4E+20GPU
GPU / 24GB
PROMPT
2129
tokens/s
GENERATION
119
tokens/s
TTFT
573
ms
LOCALSCORE
763
Apple M4 Pro 10P+4E+20GPU
GPU / 48GB
PROMPT
2133
tokens/s
GENERATION
119
tokens/s
TTFT
572
ms
LOCALSCORE
762
AMD Radeon RX 6800
GPU / 16GB
PROMPT
3322
tokens/s
GENERATION
69.0
tokens/s
TTFT
486
ms
LOCALSCORE
759
AMD Radeon RX 6650 XT
GPU / 8GB
PROMPT
2633
tokens/s
GENERATION
87.4
tokens/s
TTFT
591
ms
LOCALSCORE
734
Apple M1 Max 8P+2E+32GPU
GPU / 64GB
PROMPT
1916
tokens/s
GENERATION
115
tokens/s
TTFT
598
ms
LOCALSCORE
718
Quadro P4000
GPU / 8GB
PROMPT
2194
tokens/s
GENERATION
94.1
tokens/s
TTFT
597
ms
LOCALSCORE
702
Tesla P40
GPU / 24GB
PROMPT
2209
tokens/s
GENERATION
90.5
tokens/s
TTFT
656
ms
LOCALSCORE
700
NVIDIA GeForce GTX 1080
GPU / 8GB
PROMPT
2655
tokens/s
GENERATION
65.7
tokens/s
TTFT
668
ms
LOCALSCORE
696
Apple M4 Pro 8P+4E+16GPU
GPU / 64GB
PROMPT
1859
tokens/s
GENERATION
112
tokens/s
TTFT
670
ms
LOCALSCORE
677
Apple M4 Pro 8P+4E+16GPU
GPU / 24GB
PROMPT
1807
tokens/s
GENERATION
111
tokens/s
TTFT
674
ms
LOCALSCORE
667
AMD Radeon RX 6700 XT
GPU / 12GB
PROMPT
2883
tokens/s
GENERATION
64.3
tokens/s
TTFT
561
ms
LOCALSCORE
656
Apple M1 Max 8P+2E+24GPU
GPU / 64GB
PROMPT
1669
tokens/s
GENERATION
106
tokens/s
TTFT
689
ms
LOCALSCORE
635
Apple M4 Pro 8P+4E+16GPU
GPU / 48GB
PROMPT
1683
tokens/s
GENERATION
104
tokens/s
TTFT
743
ms
LOCALSCORE
622
Apple M1 Max 8P+2E+24GPU
GPU / 32GB
PROMPT
1582
tokens/s
GENERATION
93.9
tokens/s
TTFT
724
ms
LOCALSCORE
589
NVIDIA GeForce GTX 1060 6GB
GPU / 6GB
PROMPT
1863
tokens/s
GENERATION
76.9
tokens/s
TTFT
722
ms
LOCALSCORE
583
Apple M2 Pro 8P+4E+19GPU
GPU / 32GB
PROMPT
1488
tokens/s
GENERATION
100
tokens/s
TTFT
806
ms
LOCALSCORE
570
Apple M2 Pro 8P+4E+19GPU
GPU / 16GB
PROMPT
1458
tokens/s
GENERATION
99.5
tokens/s
TTFT
809
ms
LOCALSCORE
564
Apple M3 Pro 6P+6E+18GPU
GPU / 36GB
PROMPT
1586
tokens/s
GENERATION
89.8
tokens/s
TTFT
809
ms
LOCALSCORE
560
Apple M3 Pro 6P+6E+18GPU
GPU / 18GB
PROMPT
1574
tokens/s
GENERATION
85.6
tokens/s
TTFT
826
ms
LOCALSCORE
546
NVIDIA P106-100
GPU / 6GB
PROMPT
1758
tokens/s
GENERATION
70.3
tokens/s
TTFT
762
ms
LOCALSCORE
546
Apple M2 Pro 6P+4E+16GPU
GPU / 16GB
PROMPT
1327
tokens/s
GENERATION
91.0
tokens/s
TTFT
903
ms
LOCALSCORE
511
Apple M2 Pro 6P+4E+16GPU
GPU / 32GB
PROMPT
1281
tokens/s
GENERATION
91.5
tokens/s
TTFT
914
ms
LOCALSCORE
504
Apple M5 4P+6E+10GPU
GPU / 16GB
PROMPT
1259
tokens/s
GENERATION
99.3
tokens/s
TTFT
987
ms
LOCALSCORE
502
Apple M5 4P+6E+10GPU
GPU / 32GB
PROMPT
1275
tokens/s
GENERATION
98.2
tokens/s
TTFT
991
ms
LOCALSCORE
502
Apple M3 Pro 5P+6E+14GPU
GPU / 36GB
PROMPT
1328
tokens/s
GENERATION
88.2
tokens/s
TTFT
953
ms
LOCALSCORE
497
Apple M3 Pro 5P+6E+14GPU
GPU / 18GB
PROMPT
1293
tokens/s
GENERATION
88.4
tokens/s
TTFT
976
ms
LOCALSCORE
489
AMD Radeon RX 7600 XT
GPU / 8GB
PROMPT
1502
tokens/s
GENERATION
57.1
tokens/s
TTFT
903
ms
LOCALSCORE
456
Apple M1 Pro 8P+2E+16GPU
GPU / 16GB
PROMPT
1158
tokens/s
GENERATION
78.2
tokens/s
TTFT
1.03
sec
LOCALSCORE
445
Apple M1 Pro 8P+2E+16GPU
GPU / 32GB
PROMPT
1171
tokens/s
GENERATION
77.4
tokens/s
TTFT
1.03
sec
LOCALSCORE
445
NVIDIA GeForce RTX 2050
GPU / 4GB
PROMPT
1770
tokens/s
GENERATION
47.9
tokens/s
TTFT
967
ms
LOCALSCORE
444
AMD Radeon RX 6600
GPU / 8GB
PROMPT
1980
tokens/s
GENERATION
42.2
tokens/s
TTFT
811
ms
LOCALSCORE
437
Quadro P2000
GPU / 5GB
PROMPT
1353
tokens/s
GENERATION
58.3
tokens/s
TTFT
1.01
sec
LOCALSCORE
428
Apple M4 4P+6E+10GPU
GPU / 16GB
PROMPT
1097
tokens/s
GENERATION
76.6
tokens/s
TTFT
1.19
sec
LOCALSCORE
413
Apple M1 Pro 6P+2E+14GPU
GPU / 32GB
PROMPT
1055
tokens/s
GENERATION
71.6
tokens/s
TTFT
1.14
sec
LOCALSCORE
405
Apple M1 Pro 6P+2E+14GPU
GPU / 16GB
PROMPT
1040
tokens/s
GENERATION
71.8
tokens/s
TTFT
1.15
sec
LOCALSCORE
402
Apple M4 4P+6E+10GPU
GPU / 32GB
PROMPT
1060
tokens/s
GENERATION
75.3
tokens/s
TTFT
1.23
sec
LOCALSCORE
402
Apple M4 4P+6E+10GPU
GPU / 24GB
PROMPT
1012
tokens/s
GENERATION
73.8
tokens/s
TTFT
1.30
sec
LOCALSCORE
387
Apple M3 4P+4E+10GPU
GPU / 16GB
PROMPT
933
tokens/s
GENERATION
67.1
tokens/s
TTFT
1.41
sec
LOCALSCORE
354
Orin
GPU / 7GB
PROMPT
1157
tokens/s
GENERATION
45.3
tokens/s
TTFT
1.24
sec
LOCALSCORE
348
Apple M3 4P+4E+10GPU
GPU / 24GB
PROMPT
915
tokens/s
GENERATION
64.8
tokens/s
TTFT
1.43
sec
LOCALSCORE
346
Apple M4 4P+6E+8GPU
GPU / 16GB
PROMPT
898
tokens/s
GENERATION
65.9
tokens/s
TTFT
1.44
sec
LOCALSCORE
345
NVIDIA GeForce GTX 1050 Ti
GPU / 4GB
PROMPT
1014
tokens/s
GENERATION
45.2
tokens/s
TTFT
1.30
sec
LOCALSCORE
328
Apple M3 4P+4E+8GPU
GPU / 16GB
PROMPT
804
tokens/s
GENERATION
60.3
tokens/s
TTFT
1.62
sec
LOCALSCORE
310
Apple M2 4P+4E+10GPU
GPU / 8GB
PROMPT
826
tokens/s
GENERATION
58.2
tokens/s
TTFT
1.61
sec
LOCALSCORE
310
Apple M2 4P+4E+10GPU
GPU / 24GB
PROMPT
814
tokens/s
GENERATION
54.8
tokens/s
TTFT
1.58
sec
LOCALSCORE
304
Apple M2 4P+4E+10GPU
GPU / 16GB
PROMPT
802
tokens/s
GENERATION
55.6
tokens/s
TTFT
1.59
sec
LOCALSCORE
304
NVIDIA P106-090
GPU / 6GB
PROMPT
923
tokens/s
GENERATION
39.2
tokens/s
TTFT
1.44
sec
LOCALSCORE
293
AMD Radeon Graphics
GPU / 32GB
PROMPT
1123
tokens/s
GENERATION
44.4
tokens/s
TTFT
1.56
sec
LOCALSCORE
289
NVIDIA GeForce GTX 1660 SUPER
GPU / 6GB
PROMPT
800
tokens/s
GENERATION
35.6
tokens/s
TTFT
1.81
sec
LOCALSCORE
251
NVIDIA GeForce GTX 1050
GPU / 4GB
PROMPT
742
tokens/s
GENERATION
31.1
tokens/s
TTFT
1.71
sec
LOCALSCORE
239
NVIDIA GeForce GTX 1070 with Max-Q Design
GPU / 8GB
PROMPT
814
tokens/s
GENERATION
24.4
tokens/s
TTFT
1.83
sec
LOCALSCORE
221
Apple M1 4P+4E+8GPU
GPU / 16GB
PROMPT
584
tokens/s
GENERATION
39.8
tokens/s
TTFT
2.24
sec
LOCALSCORE
218
NVIDIA GeForce GTX 1650 SUPER
GPU / 4GB
PROMPT
667
tokens/s
GENERATION
31.2
tokens/s
TTFT
2.11
sec
LOCALSCORE
214
Apple M1 4P+4E+8GPU
GPU / 8GB
PROMPT
568
tokens/s
GENERATION
37.4
tokens/s
TTFT
2.46
sec
LOCALSCORE
208
Apple M1 4P+4E+7GPU
GPU / 8GB
PROMPT
527
tokens/s
GENERATION
38.2
tokens/s
TTFT
2.45
sec
LOCALSCORE
202
Apple M1 4P+4E+7GPU
GPU / 16GB
PROMPT
531
tokens/s
GENERATION
37.9
tokens/s
TTFT
2.45
sec
LOCALSCORE
202
Apple M2 4P+4E+8GPU
GPU / 8GB
PROMPT
523
tokens/s
GENERATION
34.5
tokens/s
TTFT
2.72
sec
LOCALSCORE
194
Apple M2 4P+4E+8GPU
GPU / 16GB
PROMPT
523
tokens/s
GENERATION
35.3
tokens/s
TTFT
3.05
sec
LOCALSCORE
193
NVIDIA GeForce GTX 1050
GPU / 2GB
PROMPT
539
tokens/s
GENERATION
27.4
tokens/s
TTFT
6.24
sec
LOCALSCORE
178
NVIDIA GeForce GT 1030
GPU / 2GB
PROMPT
543
tokens/s
GENERATION
24.0
tokens/s
TTFT
2.60
sec
LOCALSCORE
171
Apple A18 Pro 2P+4E+5GPU
GPU / 8GB
PROMPT
433
tokens/s
GENERATION
34.0
tokens/s
TTFT
3.03
sec
LOCALSCORE
169
NVIDIA GeForce GTX 1650 Ti
GPU / 4GB
PROMPT
567
tokens/s
GENERATION
20.7
tokens/s
TTFT
2.58
sec
LOCALSCORE
166
NVIDIA T1200 Laptop GPU
GPU / 4GB
PROMPT
540
tokens/s
GENERATION
21.0
tokens/s
TTFT
2.61
sec
LOCALSCORE
163
AMD Radeon 780M
GPU / 27GB
PROMPT
690
tokens/s
GENERATION
11.7
tokens/s
TTFT
1.88
sec
LOCALSCORE
162
NVIDIA GeForce GTX 1650
GPU / 4GB
PROMPT
526
tokens/s
GENERATION
22.1
tokens/s
TTFT
2.76
sec
LOCALSCORE
161
NVIDIA GeForce GTX 1650 Ti with Max-Q Design
GPU / 4GB
PROMPT
547
tokens/s
GENERATION
19.5
tokens/s
TTFT
2.62
sec
LOCALSCORE
160
AMD Radeon 780M Graphics
GPU / 14GB
PROMPT
666
tokens/s
GENERATION
11.8
tokens/s
TTFT
1.96
sec
LOCALSCORE
159
AMD Radeon 780M Graphics
GPU / 13GB
PROMPT
632
tokens/s
GENERATION
12.0
tokens/s
TTFT
2.05
sec
LOCALSCORE
155
AMD Radeon(TM) 780M
GPU / 29GB
PROMPT
617
tokens/s
GENERATION
11.5
tokens/s
TTFT
2.09
sec
LOCALSCORE
150
AMD Radeon 780M Graphics
GPU / 6GB
PROMPT
572
tokens/s
GENERATION
11.2
tokens/s
TTFT
2.32
sec
LOCALSCORE
140
AMD Radeon(TM) Graphics
GPU / 25GB
PROMPT
565
tokens/s
GENERATION
10.7
tokens/s
TTFT
2.29
sec
LOCALSCORE
138
AMD Radeon 760M Graphics
GPU / 12GB
PROMPT
480
tokens/s
GENERATION
10.4
tokens/s
TTFT
2.81
sec
LOCALSCORE
121
AMD RadeonT 780M
GPU / 13GB
PROMPT
506
tokens/s
GENERATION
7.5
tokens/s
TTFT
2.64
sec
LOCALSCORE
113
Quadro P620
GPU / 2GB
PROMPT
330
tokens/s
GENERATION
18.6
tokens/s
TTFT
5.30
sec
LOCALSCORE
105
NVIDIA GeForce MX150
GPU / 2GB
PROMPT
311
tokens/s
GENERATION
13.5
tokens/s
TTFT
4.06
sec
LOCALSCORE
101
Radeon RX 5500 XT
GPU / 4GB
PROMPT
174
tokens/s
GENERATION
56.2
tokens/s
TTFT
11.45
sec
LOCALSCORE
95
AMD Radeon 780M Graphics
GPU / 19GB
PROMPT
399
tokens/s
GENERATION
5.9
tokens/s
TTFT
3.36
sec
LOCALSCORE
89
AMD Radeon 740M Graphics
GPU / 12GB
PROMPT
212
tokens/s
GENERATION
6.2
tokens/s
TTFT
6.32
sec
LOCALSCORE
59