T5La-Large

This model is a fine-tuned version of on the HuggingFaceFW/fineweb sample-350BT dataset. It achieves the following results on the evaluation set:

• Loss: 6.3929
• Accuracy: 0.0409

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• training_steps: 524288
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
6.9745	0.0095	5000	6.9068	0.0314
6.7778	0.0191	10000	6.7410	0.0313
6.715	0.0286	15000	6.6431	0.0333
6.6547	0.0381	20000	6.5784	0.0335
6.5695	0.0477	25000	6.5298	0.0341
6.5283	0.0572	30000	6.5265	0.0340
6.5826	0.0668	35000	6.4902	0.0340
6.5631	0.0763	40000	6.4705	0.0357
6.5234	0.0858	45000	6.4617	0.0339
6.548	0.0954	50000	6.4475	0.0350
6.599	0.1049	55000	6.4514	0.0355
6.547	0.1144	60000	6.4568	0.0364
6.5265	0.1240	65000	6.4709	0.0381
6.5916	0.1335	70000	6.4962	0.0372
6.5919	0.1431	75000	6.4827	0.0375
6.4745	0.1526	80000	6.4902	0.0385
6.6695	0.1621	85000	6.5442	0.0384
6.5994	0.1717	90000	6.5036	0.0387
6.6023	0.1812	95000	6.4979	0.0385
6.5913	0.1907	100000	6.5830	0.0382
6.6542	0.2003	105000	6.5550	0.0384
6.6602	0.2098	110000	6.5538	0.0388
6.6113	0.2193	115000	6.5508	0.0393
6.6568	0.2289	120000	6.5527	0.0388
6.6476	0.2384	125000	6.5457	0.0391
6.6636	0.2480	130000	6.5458	0.0401
6.6318	0.2575	135000	6.5468	0.0399
6.6358	0.2670	140000	6.5729	0.0374
6.639	0.2766	145000	6.5532	0.0394
6.6817	0.2861	150000	6.5516	0.0391
6.6339	0.2956	155000	6.5509	0.0389
6.6132	0.3052	160000	6.5391	0.0381
6.6347	0.3147	165000	6.5376	0.0386
6.6542	0.3242	170000	6.5377	0.0397
6.619	0.3338	175000	6.5441	0.0388
6.6979	0.3433	180000	6.5556	0.0394
6.6485	0.3529	185000	6.5370	0.0399
6.6035	0.3624	190000	6.5300	0.0389
6.6574	0.3719	195000	6.5272	0.0385
6.6152	0.3815	200000	6.5326	0.0377
6.5946	0.3910	205000	6.5314	0.0386
6.6747	0.4005	210000	6.5184	0.0390
6.618	0.4101	215000	6.5262	0.0375
6.6218	0.4196	220000	6.5339	0.0388
6.6659	0.4292	225000	6.5258	0.0383
6.6292	0.4387	230000	6.5242	0.0387
6.6608	0.4482	235000	6.5588	0.0359
6.5772	0.4578	240000	6.5117	0.0389
6.5961	0.4673	245000	6.5381	0.0359
6.563	0.4768	250000	6.5191	0.0401
6.5651	0.4864	255000	6.4980	0.0385
6.5398	0.4959	260000	6.4998	0.0389
6.5368	0.5054	265000	6.4984	0.0389
6.599	0.5150	270000	6.4935	0.0388
6.6015	0.5245	275000	6.4891	0.0388
6.5597	0.5341	280000	6.4779	0.0394
6.5695	0.5436	285000	6.4823	0.0395
6.5809	0.5531	290000	6.4925	0.0382
6.6522	0.5627	295000	6.4898	0.0390
6.5688	0.5722	300000	6.4914	0.0389
6.5445	0.5817	305000	6.4828	0.0389
6.5674	0.5913	310000	6.4940	0.0388
6.5926	0.6008	315000	6.4786	0.0388
6.4979	0.6104	320000	6.4754	0.0391
6.5669	0.6199	325000	6.4722	0.0394
6.5335	0.6294	330000	6.4731	0.0398
6.5727	0.6390	335000	6.4770	0.0394
6.5735	0.6485	340000	6.4814	0.0390
6.5672	0.6580	345000	6.4611	0.0394
6.5223	0.6676	350000	6.4624	0.0397
6.518	0.6771	355000	6.4734	0.0402
6.5466	0.6866	360000	6.4640	0.0399
6.5213	0.6962	365000	6.4624	0.0401
6.5881	0.7057	370000	6.4505	0.0399
6.5353	0.7153	375000	6.4470	0.0404
6.5371	0.7248	380000	6.4453	0.0401
6.5153	0.7343	385000	6.4461	0.0401
6.5488	0.7439	390000	6.4434	0.0401
6.5121	0.7534	395000	6.4468	0.0395
6.5106	0.7629	400000	6.4364	0.0399
6.5572	0.7725	405000	6.4443	0.0405
6.494	0.7820	410000	6.4338	0.0402
6.5233	0.7915	415000	6.4318	0.0404
6.4718	0.8011	420000	6.4255	0.0415
6.4915	0.8106	425000	6.4261	0.0402
6.5007	0.8202	430000	6.4244	0.0407
6.488	0.8297	435000	6.4187	0.0405
6.5093	0.8392	440000	6.4163	0.0402
6.5029	0.8488	445000	6.4152	0.0403
6.3943	0.8583	450000	6.4201	0.0402
6.5358	0.8678	455000	6.4104	0.0410
6.5185	0.8774	460000	6.4099	0.0411
6.4622	0.8869	465000	6.4110	0.0408
6.4632	0.8965	470000	6.4088	0.0405
6.5168	0.9060	475000	6.4056	0.0408
6.4607	0.9155	480000	6.4000	0.0404
6.4444	0.9251	485000	6.4024	0.0404
6.5035	0.9346	490000	6.3991	0.0404
6.4861	0.9441	495000	6.3996	0.0407
6.4772	0.9537	500000	6.3965	0.0403
6.4699	1.0095	505000	6.3943	0.0408
6.4423	1.0191	510000	6.3959	0.0407
6.4724	1.0286	515000	6.3930	0.0410
6.4827	1.0381	520000	6.3910	0.0409

Framework versions

• Transformers 4.56.0.dev0
• Pytorch 2.8.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4