model icon

Llama 3.2 1B Instruct

Q4_K - Medium

1.5Bparams
COMPARE ACCELERATORS

300 accelerators tested

Select Accelerators

NVIDIA RTX PRO 6000 Blackwell Workstation Edition

95GB

NVIDIA RTX 6000 Ada Generation

47GB

NVIDIA GeForce RTX 4090 D

47GB

NVIDIA H100 PCIe

79GB

NVIDIA A100-SXM4-80GB

79GB

Llama 3.2 1B Instruct - Q4_K - Medium

LEADERBOARD
PROMPT
30896
tokens/s
GENERATION
374
tokens/s
TTFT
43
ms
LOCALSCORE
6431
PROMPT
27892
tokens/s
GENERATION
416
tokens/s
TTFT
53
ms
LOCALSCORE
6030
PROMPT
24549
tokens/s
GENERATION
359
tokens/s
TTFT
57
ms
LOCALSCORE
5380
PROMPT
18590
tokens/s
GENERATION
335
tokens/s
TTFT
68
ms
LOCALSCORE
4512
PROMPT
15370
tokens/s
GENERATION
308
tokens/s
TTFT
80
ms
LOCALSCORE
3892
PROMPT
14892
tokens/s
GENERATION
345
tokens/s
TTFT
89
ms
LOCALSCORE
3863
PROMPT
14352
tokens/s
GENERATION
326
tokens/s
TTFT
93
ms
LOCALSCORE
3688
PROMPT
13191
tokens/s
GENERATION
315
tokens/s
TTFT
102
ms
LOCALSCORE
3438
PROMPT
19620
tokens/s
GENERATION
131
tokens/s
TTFT
68
ms
LOCALSCORE
3350
PROMPT
15512
tokens/s
GENERATION
226
tokens/s
TTFT
93
ms
LOCALSCORE
3334
PROMPT
12044
tokens/s
GENERATION
312
tokens/s
TTFT
110
ms
LOCALSCORE
3243
PROMPT
24526
tokens/s
GENERATION
217
tokens/s
TTFT
755
ms
LOCALSCORE
3205
PROMPT
12717
tokens/s
GENERATION
283
tokens/s
TTFT
112
ms
LOCALSCORE
3179
PROMPT
13131
tokens/s
GENERATION
256
tokens/s
TTFT
103
ms
LOCALSCORE
3163
GPU / 44GB
PROMPT
12156
tokens/s
GENERATION
284
tokens/s
TTFT
113
ms
LOCALSCORE
3126
GPU / 45GB
PROMPT
18255
tokens/s
GENERATION
121
tokens/s
TTFT
75
ms
LOCALSCORE
3088
PROMPT
16736
tokens/s
GENERATION
141
tokens/s
TTFT
81
ms
LOCALSCORE
3077
PROMPT
16850
tokens/s
GENERATION
159
tokens/s
TTFT
116
ms
LOCALSCORE
3038
GPU / 44GB
PROMPT
17686
tokens/s
GENERATION
118
tokens/s
TTFT
77
ms
LOCALSCORE
3000
PROMPT
13347
tokens/s
GENERATION
188
tokens/s
TTFT
106
ms
LOCALSCORE
2856
PROMPT
13578
tokens/s
GENERATION
190
tokens/s
TTFT
112
ms
LOCALSCORE
2836
PROMPT
10328
tokens/s
GENERATION
285
tokens/s
TTFT
131
ms
LOCALSCORE
2820
PROMPT
10074
tokens/s
GENERATION
297
tokens/s
TTFT
137
ms
LOCALSCORE
2795
PROMPT
11541
tokens/s
GENERATION
206
tokens/s
TTFT
118
ms
LOCALSCORE
2711
PROMPT
9892
tokens/s
GENERATION
249
tokens/s
TTFT
129
ms
LOCALSCORE
2672
PROMPT
9743
tokens/s
GENERATION
250
tokens/s
TTFT
131
ms
LOCALSCORE
2647
PROMPT
10703
tokens/s
GENERATION
237
tokens/s
TTFT
142
ms
LOCALSCORE
2616
GPU / 47GB
PROMPT
9356
tokens/s
GENERATION
250
tokens/s
TTFT
141
ms
LOCALSCORE
2550
PROMPT
8984
tokens/s
GENERATION
248
tokens/s
TTFT
135
ms
LOCALSCORE
2545
PROMPT
9991
tokens/s
GENERATION
236
tokens/s
TTFT
148
ms
LOCALSCORE
2515
GPU / 140GB
PROMPT
14389
tokens/s
GENERATION
98.4
tokens/s
TTFT
91
ms
LOCALSCORE
2498
PROMPT
14608
tokens/s
GENERATION
96.2
tokens/s
TTFT
91
ms
LOCALSCORE
2490
PROMPT
13606
tokens/s
GENERATION
103
tokens/s
TTFT
96
ms
LOCALSCORE
2447
PROMPT
16286
tokens/s
GENERATION
239
tokens/s
TTFT
728
ms
LOCALSCORE
2447
PROMPT
8404
tokens/s
GENERATION
241
tokens/s
TTFT
168
ms
LOCALSCORE
2296
PROMPT
8503
tokens/s
GENERATION
230
tokens/s
TTFT
165
ms
LOCALSCORE
2277
PROMPT
8626
tokens/s
GENERATION
210
tokens/s
TTFT
153
ms
LOCALSCORE
2276
PROMPT
9203
tokens/s
GENERATION
209
tokens/s
TTFT
168
ms
LOCALSCORE
2254
GPU / 12GB
PROMPT
7853
tokens/s
GENERATION
232
tokens/s
TTFT
159
ms
LOCALSCORE
2252
GPU / 93GB
PROMPT
13164
tokens/s
GENERATION
83.9
tokens/s
TTFT
99
ms
LOCALSCORE
2232
GPU / 22GB
PROMPT
9075
tokens/s
GENERATION
189
tokens/s
TTFT
171
ms
LOCALSCORE
2157
PROMPT
8206
tokens/s
GENERATION
181
tokens/s
TTFT
161
ms
LOCALSCORE
2099
PROMPT
8737
tokens/s
GENERATION
189
tokens/s
TTFT
179
ms
LOCALSCORE
2099
PROMPT
11035
tokens/s
GENERATION
130
tokens/s
TTFT
179
ms
LOCALSCORE
2001
PROMPT
7588
tokens/s
GENERATION
184
tokens/s
TTFT
181
ms
LOCALSCORE
1972
PROMPT
10806
tokens/s
GENERATION
81.9
tokens/s
TTFT
117
ms
LOCALSCORE
1964
PROMPT
7201
tokens/s
GENERATION
190
tokens/s
TTFT
208
ms
LOCALSCORE
1874
PROMPT
6350
tokens/s
GENERATION
212
tokens/s
TTFT
216
ms
LOCALSCORE
1840
GPU / 22GB
PROMPT
8777
tokens/s
GENERATION
93.5
tokens/s
TTFT
153
ms
LOCALSCORE
1750
PROMPT
6064
tokens/s
GENERATION
198
tokens/s
TTFT
232
ms
LOCALSCORE
1731
PROMPT
6087
tokens/s
GENERATION
168
tokens/s
TTFT
251
ms
LOCALSCORE
1599
PROMPT
5553
tokens/s
GENERATION
195
tokens/s
TTFT
287
ms
LOCALSCORE
1558
PROMPT
6225
tokens/s
GENERATION
96.4
tokens/s
TTFT
224
ms
LOCALSCORE
1388
PROMPT
7267
tokens/s
GENERATION
70.5
tokens/s
TTFT
191
ms
LOCALSCORE
1388
PROMPT
3925
tokens/s
GENERATION
181
tokens/s
TTFT
302
ms
LOCALSCORE
1330
GPU / 12GB
PROMPT
4298
tokens/s
GENERATION
160
tokens/s
TTFT
301
ms
LOCALSCORE
1317
PROMPT
3796
tokens/s
GENERATION
183
tokens/s
TTFT
307
ms
LOCALSCORE
1314
PROMPT
8328
tokens/s
GENERATION
101
tokens/s
TTFT
371
ms
LOCALSCORE
1312
PROMPT
4979
tokens/s
GENERATION
114
tokens/s
TTFT
286
ms
LOCALSCORE
1255
PROMPT
4652
tokens/s
GENERATION
121
tokens/s
TTFT
287
ms
LOCALSCORE
1252
PROMPT
4363
tokens/s
GENERATION
144
tokens/s
TTFT
330
ms
LOCALSCORE
1241
PROMPT
4061
tokens/s
GENERATION
145
tokens/s
TTFT
318
ms
LOCALSCORE
1227
PROMPT
3296
tokens/s
GENERATION
176
tokens/s
TTFT
334
ms
LOCALSCORE
1203
PROMPT
3272
tokens/s
GENERATION
170
tokens/s
TTFT
339
ms
LOCALSCORE
1179
PROMPT
4087
tokens/s
GENERATION
130
tokens/s
TTFT
354
ms
LOCALSCORE
1144
PROMPT
3211
tokens/s
GENERATION
169
tokens/s
TTFT
371
ms
LOCALSCORE
1136
PROMPT
3292
tokens/s
GENERATION
146
tokens/s
TTFT
358
ms
LOCALSCORE
1104
PROMPT
6502
tokens/s
GENERATION
34.6
tokens/s
TTFT
216
ms
LOCALSCORE
1013
PROMPT
2708
tokens/s
GENERATION
163
tokens/s
TTFT
461
ms
LOCALSCORE
984
PROMPT
3399
tokens/s
GENERATION
109
tokens/s
TTFT
416
ms
LOCALSCORE
962
PROMPT
2582
tokens/s
GENERATION
138
tokens/s
TTFT
425
ms
LOCALSCORE
943
PROMPT
4161
tokens/s
GENERATION
83.8
tokens/s
TTFT
348
ms
LOCALSCORE
940
PROMPT
2980
tokens/s
GENERATION
116
tokens/s
TTFT
441
ms
LOCALSCORE
922
PROMPT
3355
tokens/s
GENERATION
98.0
tokens/s
TTFT
423
ms
LOCALSCORE
920
PROMPT
3084
tokens/s
GENERATION
105
tokens/s
TTFT
431
ms
LOCALSCORE
910
PROMPT
4042
tokens/s
GENERATION
74.7
tokens/s
TTFT
354
ms
LOCALSCORE
898
PROMPT
2520
tokens/s
GENERATION
133
tokens/s
TTFT
476
ms
LOCALSCORE
889
PROMPT
2516
tokens/s
GENERATION
124
tokens/s
TTFT
477
ms
LOCALSCORE
868
PROMPT
2512
tokens/s
GENERATION
121
tokens/s
TTFT
490
ms
LOCALSCORE
852
PROMPT
4411
tokens/s
GENERATION
62.8
tokens/s
TTFT
409
ms
LOCALSCORE
849
PROMPT
3955
tokens/s
GENERATION
60.9
tokens/s
TTFT
444
ms
LOCALSCORE
783
PROMPT
2038
tokens/s
GENERATION
129
tokens/s
TTFT
554
ms
LOCALSCORE
781
PROMPT
2060
tokens/s
GENERATION
124
tokens/s
TTFT
552
ms
LOCALSCORE
774
PROMPT
2138
tokens/s
GENERATION
121
tokens/s
TTFT
568
ms
LOCALSCORE
770
PROMPT
1991
tokens/s
GENERATION
125
tokens/s
TTFT
565
ms
LOCALSCORE
761
PROMPT
2128
tokens/s
GENERATION
118
tokens/s
TTFT
580
ms
LOCALSCORE
756
PROMPT
2378
tokens/s
GENERATION
97.0
tokens/s
TTFT
543
ms
LOCALSCORE
752
PROMPT
2071
tokens/s
GENERATION
116
tokens/s
TTFT
590
ms
LOCALSCORE
741
PROMPT
2633
tokens/s
GENERATION
87.4
tokens/s
TTFT
591
ms
LOCALSCORE
734
PROMPT
2933
tokens/s
GENERATION
69.4
tokens/s
TTFT
548
ms
LOCALSCORE
689
PROMPT
3196
tokens/s
GENERATION
55.7
tokens/s
TTFT
524
ms
LOCALSCORE
668
PROMPT
1791
tokens/s
GENERATION
111
tokens/s
TTFT
676
ms
LOCALSCORE
664
PROMPT
1735
tokens/s
GENERATION
111
tokens/s
TTFT
683
ms
LOCALSCORE
655
PROMPT
1684
tokens/s
GENERATION
106
tokens/s
TTFT
682
ms
LOCALSCORE
640
PROMPT
2052
tokens/s
GENERATION
72.9
tokens/s
TTFT
768
ms
LOCALSCORE
580
PROMPT
1486
tokens/s
GENERATION
99.0
tokens/s
TTFT
811
ms
LOCALSCORE
566
PROMPT
1556
tokens/s
GENERATION
89.6
tokens/s
TTFT
816
ms
LOCALSCORE
555
PROMPT
1344
tokens/s
GENERATION
88.1
tokens/s
TTFT
959
ms
LOCALSCORE
498
PROMPT
1328
tokens/s
GENERATION
88.2
tokens/s
TTFT
953
ms
LOCALSCORE
497
PROMPT
1502
tokens/s
GENERATION
57.1
tokens/s
TTFT
903
ms
LOCALSCORE
456
PROMPT
1770
tokens/s
GENERATION
47.9
tokens/s
TTFT
967
ms
LOCALSCORE
444
PROMPT
1166
tokens/s
GENERATION
77.2
tokens/s
TTFT
1.04
sec
LOCALSCORE
443
PROMPT
1098
tokens/s
GENERATION
76.1
tokens/s
TTFT
1.19
sec
LOCALSCORE
412
PROMPT
1064
tokens/s
GENERATION
71.0
tokens/s
TTFT
1.13
sec
LOCALSCORE
405
PROMPT
1033
tokens/s
GENERATION
71.6
tokens/s
TTFT
1.16
sec
LOCALSCORE
400
PROMPT
1028
tokens/s
GENERATION
74.6
tokens/s
TTFT
1.25
sec
LOCALSCORE
395
GPU / 7GB
PROMPT
1233
tokens/s
GENERATION
47.8
tokens/s
TTFT
1.18
sec
LOCALSCORE
369
PROMPT
938
tokens/s
GENERATION
67.8
tokens/s
TTFT
1.41
sec
LOCALSCORE
356
PROMPT
914
tokens/s
GENERATION
64.1
tokens/s
TTFT
1.43
sec
LOCALSCORE
345
PROMPT
1057
tokens/s
GENERATION
46.9
tokens/s
TTFT
1.22
sec
LOCALSCORE
343
PROMPT
1013
tokens/s
GENERATION
44.5
tokens/s
TTFT
1.33
sec
LOCALSCORE
324
PROMPT
827
tokens/s
GENERATION
56.9
tokens/s
TTFT
1.57
sec
LOCALSCORE
310
PROMPT
805
tokens/s
GENERATION
56.5
tokens/s
TTFT
1.67
sec
LOCALSCORE
301
PROMPT
780
tokens/s
GENERATION
51.6
tokens/s
TTFT
1.60
sec
LOCALSCORE
293
PROMPT
792
tokens/s
GENERATION
35.0
tokens/s
TTFT
1.58
sec
LOCALSCORE
260
PROMPT
611
tokens/s
GENERATION
40.2
tokens/s
TTFT
2.15
sec
LOCALSCORE
225
PROMPT
814
tokens/s
GENERATION
24.4
tokens/s
TTFT
1.83
sec
LOCALSCORE
221
PROMPT
589
tokens/s
GENERATION
40.0
tokens/s
TTFT
2.25
sec
LOCALSCORE
219
PROMPT
538
tokens/s
GENERATION
38.5
tokens/s
TTFT
2.41
sec
LOCALSCORE
205
PROMPT
567
tokens/s
GENERATION
20.7
tokens/s
TTFT
2.58
sec
LOCALSCORE
166
PROMPT
540
tokens/s
GENERATION
21.0
tokens/s
TTFT
2.61
sec
LOCALSCORE
163
GPU / 27GB
PROMPT
690
tokens/s
GENERATION
11.7
tokens/s
TTFT
1.88
sec
LOCALSCORE
162
PROMPT
666
tokens/s
GENERATION
11.8
tokens/s
TTFT
1.96
sec
LOCALSCORE
159
PROMPT
429
tokens/s
GENERATION
24.6
tokens/s
TTFT
3.27
sec
LOCALSCORE
148
PROMPT
311
tokens/s
GENERATION
13.5
tokens/s
TTFT
4.06
sec
LOCALSCORE
101