model icon

Llama 3.2 1B Instruct

Q4_K - Medium

1.5Bparams
COMPARE ACCELERATORS

224 accelerators tested

Select Accelerators

NVIDIA RTX 6000 Ada Generation

47GB

NVIDIA H100 PCIe

79GB

NVIDIA A100-SXM4-80GB

79GB

NVIDIA GeForce RTX 3090 Ti

24GB

NVIDIA GeForce RTX 3080 Ti

12GB

Llama 3.2 1B Instruct - Q4_K - Medium

LEADERBOARD
PROMPT
27892
tokens/s
GENERATION
416
tokens/s
TTFT
53
ms
LOCALSCORE
6030
PROMPT
18590
tokens/s
GENERATION
335
tokens/s
TTFT
68
ms
LOCALSCORE
4512
PROMPT
15370
tokens/s
GENERATION
308
tokens/s
TTFT
80
ms
LOCALSCORE
3892
PROMPT
14892
tokens/s
GENERATION
345
tokens/s
TTFT
89
ms
LOCALSCORE
3863
PROMPT
14352
tokens/s
GENERATION
326
tokens/s
TTFT
93
ms
LOCALSCORE
3688
PROMPT
13191
tokens/s
GENERATION
315
tokens/s
TTFT
102
ms
LOCALSCORE
3438
PROMPT
19620
tokens/s
GENERATION
131
tokens/s
TTFT
68
ms
LOCALSCORE
3350
PROMPT
18107
tokens/s
GENERATION
182
tokens/s
TTFT
113
ms
LOCALSCORE
3344
PROMPT
15512
tokens/s
GENERATION
226
tokens/s
TTFT
93
ms
LOCALSCORE
3334
PROMPT
12044
tokens/s
GENERATION
312
tokens/s
TTFT
110
ms
LOCALSCORE
3243
PROMPT
24526
tokens/s
GENERATION
217
tokens/s
TTFT
755
ms
LOCALSCORE
3205
PROMPT
12717
tokens/s
GENERATION
283
tokens/s
TTFT
112
ms
LOCALSCORE
3179
GPU / 44GB
PROMPT
12108
tokens/s
GENERATION
283
tokens/s
TTFT
113
ms
LOCALSCORE
3117
GPU / 45GB
PROMPT
18255
tokens/s
GENERATION
121
tokens/s
TTFT
75
ms
LOCALSCORE
3088
PROMPT
13127
tokens/s
GENERATION
240
tokens/s
TTFT
103
ms
LOCALSCORE
3086
PROMPT
16736
tokens/s
GENERATION
141
tokens/s
TTFT
81
ms
LOCALSCORE
3077
GPU / 44GB
PROMPT
17686
tokens/s
GENERATION
118
tokens/s
TTFT
77
ms
LOCALSCORE
3000
PROMPT
13347
tokens/s
GENERATION
188
tokens/s
TTFT
106
ms
LOCALSCORE
2856
PROMPT
13578
tokens/s
GENERATION
190
tokens/s
TTFT
112
ms
LOCALSCORE
2836
PROMPT
10328
tokens/s
GENERATION
285
tokens/s
TTFT
131
ms
LOCALSCORE
2820
PROMPT
10074
tokens/s
GENERATION
297
tokens/s
TTFT
137
ms
LOCALSCORE
2795
PROMPT
11444
tokens/s
GENERATION
206
tokens/s
TTFT
119
ms
LOCALSCORE
2692
PROMPT
9743
tokens/s
GENERATION
250
tokens/s
TTFT
131
ms
LOCALSCORE
2647
PROMPT
10703
tokens/s
GENERATION
237
tokens/s
TTFT
142
ms
LOCALSCORE
2616
GPU / 47GB
PROMPT
9356
tokens/s
GENERATION
250
tokens/s
TTFT
141
ms
LOCALSCORE
2550
PROMPT
8984
tokens/s
GENERATION
248
tokens/s
TTFT
135
ms
LOCALSCORE
2545
GPU / 140GB
PROMPT
14389
tokens/s
GENERATION
98.4
tokens/s
TTFT
91
ms
LOCALSCORE
2498
PROMPT
14608
tokens/s
GENERATION
96.2
tokens/s
TTFT
91
ms
LOCALSCORE
2490
PROMPT
13606
tokens/s
GENERATION
103
tokens/s
TTFT
96
ms
LOCALSCORE
2447
PROMPT
16286
tokens/s
GENERATION
239
tokens/s
TTFT
728
ms
LOCALSCORE
2447
PROMPT
8423
tokens/s
GENERATION
242
tokens/s
TTFT
167
ms
LOCALSCORE
2305
PROMPT
8404
tokens/s
GENERATION
241
tokens/s
TTFT
168
ms
LOCALSCORE
2296
PROMPT
9274
tokens/s
GENERATION
217
tokens/s
TTFT
166
ms
LOCALSCORE
2294
PROMPT
8626
tokens/s
GENERATION
210
tokens/s
TTFT
153
ms
LOCALSCORE
2276
GPU / 12GB
PROMPT
7853
tokens/s
GENERATION
232
tokens/s
TTFT
159
ms
LOCALSCORE
2252
GPU / 93GB
PROMPT
13164
tokens/s
GENERATION
83.9
tokens/s
TTFT
99
ms
LOCALSCORE
2232
GPU / 22GB
PROMPT
9075
tokens/s
GENERATION
189
tokens/s
TTFT
171
ms
LOCALSCORE
2157
PROMPT
8206
tokens/s
GENERATION
181
tokens/s
TTFT
161
ms
LOCALSCORE
2099
PROMPT
8737
tokens/s
GENERATION
189
tokens/s
TTFT
179
ms
LOCALSCORE
2099
PROMPT
11035
tokens/s
GENERATION
130
tokens/s
TTFT
179
ms
LOCALSCORE
2001
PROMPT
7588
tokens/s
GENERATION
184
tokens/s
TTFT
181
ms
LOCALSCORE
1972
PROMPT
10806
tokens/s
GENERATION
81.9
tokens/s
TTFT
117
ms
LOCALSCORE
1964
PROMPT
7201
tokens/s
GENERATION
190
tokens/s
TTFT
208
ms
LOCALSCORE
1874
PROMPT
6350
tokens/s
GENERATION
212
tokens/s
TTFT
216
ms
LOCALSCORE
1840
PROMPT
6092
tokens/s
GENERATION
208
tokens/s
TTFT
232
ms
LOCALSCORE
1760
GPU / 22GB
PROMPT
8777
tokens/s
GENERATION
93.5
tokens/s
TTFT
153
ms
LOCALSCORE
1750
PROMPT
6087
tokens/s
GENERATION
168
tokens/s
TTFT
251
ms
LOCALSCORE
1599
PROMPT
5553
tokens/s
GENERATION
195
tokens/s
TTFT
287
ms
LOCALSCORE
1558
PROMPT
6225
tokens/s
GENERATION
96.4
tokens/s
TTFT
224
ms
LOCALSCORE
1388
PROMPT
7267
tokens/s
GENERATION
70.5
tokens/s
TTFT
191
ms
LOCALSCORE
1388
PROMPT
3925
tokens/s
GENERATION
181
tokens/s
TTFT
302
ms
LOCALSCORE
1330
GPU / 12GB
PROMPT
4298
tokens/s
GENERATION
160
tokens/s
TTFT
301
ms
LOCALSCORE
1317
PROMPT
3780
tokens/s
GENERATION
184
tokens/s
TTFT
307
ms
LOCALSCORE
1313
PROMPT
8328
tokens/s
GENERATION
101
tokens/s
TTFT
371
ms
LOCALSCORE
1312
PROMPT
4979
tokens/s
GENERATION
114
tokens/s
TTFT
286
ms
LOCALSCORE
1255
PROMPT
4363
tokens/s
GENERATION
144
tokens/s
TTFT
330
ms
LOCALSCORE
1241
PROMPT
4047
tokens/s
GENERATION
144
tokens/s
TTFT
320
ms
LOCALSCORE
1221
PROMPT
3296
tokens/s
GENERATION
176
tokens/s
TTFT
334
ms
LOCALSCORE
1203
PROMPT
3272
tokens/s
GENERATION
170
tokens/s
TTFT
339
ms
LOCALSCORE
1179
PROMPT
3211
tokens/s
GENERATION
169
tokens/s
TTFT
371
ms
LOCALSCORE
1136
PROMPT
4083
tokens/s
GENERATION
124
tokens/s
TTFT
353
ms
LOCALSCORE
1128
PROMPT
3292
tokens/s
GENERATION
146
tokens/s
TTFT
358
ms
LOCALSCORE
1104
PROMPT
6502
tokens/s
GENERATION
34.6
tokens/s
TTFT
216
ms
LOCALSCORE
1013
PROMPT
2708
tokens/s
GENERATION
163
tokens/s
TTFT
461
ms
LOCALSCORE
984
PROMPT
3399
tokens/s
GENERATION
109
tokens/s
TTFT
416
ms
LOCALSCORE
962
PROMPT
2582
tokens/s
GENERATION
138
tokens/s
TTFT
425
ms
LOCALSCORE
943
PROMPT
4161
tokens/s
GENERATION
83.8
tokens/s
TTFT
348
ms
LOCALSCORE
940
PROMPT
2980
tokens/s
GENERATION
116
tokens/s
TTFT
441
ms
LOCALSCORE
922
PROMPT
2516
tokens/s
GENERATION
124
tokens/s
TTFT
477
ms
LOCALSCORE
868
PROMPT
4411
tokens/s
GENERATION
62.8
tokens/s
TTFT
409
ms
LOCALSCORE
849
PROMPT
3054
tokens/s
GENERATION
92.5
tokens/s
TTFT
510
ms
LOCALSCORE
821
PROMPT
3955
tokens/s
GENERATION
60.9
tokens/s
TTFT
444
ms
LOCALSCORE
783
PROMPT
2060
tokens/s
GENERATION
124
tokens/s
TTFT
552
ms
LOCALSCORE
774
PROMPT
2126
tokens/s
GENERATION
119
tokens/s
TTFT
569
ms
LOCALSCORE
763
PROMPT
2378
tokens/s
GENERATION
97.0
tokens/s
TTFT
543
ms
LOCALSCORE
752
PROMPT
2071
tokens/s
GENERATION
116
tokens/s
TTFT
590
ms
LOCALSCORE
741
PROMPT
2633
tokens/s
GENERATION
87.4
tokens/s
TTFT
591
ms
LOCALSCORE
734
PROMPT
1846
tokens/s
GENERATION
111
tokens/s
TTFT
669
ms
LOCALSCORE
674
PROMPT
3196
tokens/s
GENERATION
55.7
tokens/s
TTFT
524
ms
LOCALSCORE
668
PROMPT
1735
tokens/s
GENERATION
111
tokens/s
TTFT
683
ms
LOCALSCORE
655
PROMPT
1477
tokens/s
GENERATION
98.0
tokens/s
TTFT
819
ms
LOCALSCORE
561
PROMPT
1556
tokens/s
GENERATION
89.6
tokens/s
TTFT
816
ms
LOCALSCORE
555
PROMPT
1770
tokens/s
GENERATION
47.9
tokens/s
TTFT
967
ms
LOCALSCORE
444
PROMPT
1154
tokens/s
GENERATION
76.3
tokens/s
TTFT
1.05
sec
LOCALSCORE
438
PROMPT
1106
tokens/s
GENERATION
76.5
tokens/s
TTFT
1.18
sec
LOCALSCORE
415
PROMPT
1067
tokens/s
GENERATION
72.8
tokens/s
TTFT
1.12
sec
LOCALSCORE
411
PROMPT
1064
tokens/s
GENERATION
71.0
tokens/s
TTFT
1.13
sec
LOCALSCORE
405
PROMPT
1028
tokens/s
GENERATION
74.6
tokens/s
TTFT
1.25
sec
LOCALSCORE
395
GPU / 7GB
PROMPT
1233
tokens/s
GENERATION
47.8
tokens/s
TTFT
1.18
sec
LOCALSCORE
369
PROMPT
938
tokens/s
GENERATION
67.7
tokens/s
TTFT
1.41
sec
LOCALSCORE
356
PROMPT
938
tokens/s
GENERATION
67.8
tokens/s
TTFT
1.41
sec
LOCALSCORE
356
PROMPT
1057
tokens/s
GENERATION
46.9
tokens/s
TTFT
1.22
sec
LOCALSCORE
343
PROMPT
835
tokens/s
GENERATION
60.0
tokens/s
TTFT
1.55
sec
LOCALSCORE
318
PROMPT
827
tokens/s
GENERATION
59.9
tokens/s
TTFT
1.56
sec
LOCALSCORE
317
PROMPT
792
tokens/s
GENERATION
35.0
tokens/s
TTFT
1.58
sec
LOCALSCORE
260
PROMPT
615
tokens/s
GENERATION
40.7
tokens/s
TTFT
2.11
sec
LOCALSCORE
228
PROMPT
814
tokens/s
GENERATION
24.4
tokens/s
TTFT
1.83
sec
LOCALSCORE
221
PROMPT
589
tokens/s
GENERATION
40.0
tokens/s
TTFT
2.25
sec
LOCALSCORE
219
PROMPT
540
tokens/s
GENERATION
38.4
tokens/s
TTFT
2.40
sec
LOCALSCORE
205
PROMPT
567
tokens/s
GENERATION
20.7
tokens/s
TTFT
2.58
sec
LOCALSCORE
166
GPU / 27GB
PROMPT
690
tokens/s
GENERATION
11.7
tokens/s
TTFT
1.88
sec
LOCALSCORE
162
PROMPT
429
tokens/s
GENERATION
24.6
tokens/s
TTFT
3.27
sec
LOCALSCORE
148
PROMPT
311
tokens/s
GENERATION
13.5
tokens/s
TTFT
4.06
sec
LOCALSCORE
101