3: W0902 18:58:55.388000 284480 torch/distributed/run.py:792] 
3: W0902 18:58:55.388000 284480 torch/distributed/run.py:792] *****************************************
3: W0902 18:58:55.388000 284480 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
3: W0902 18:58:55.388000 284480 torch/distributed/run.py:792] *****************************************
0: W0902 18:58:55.639000 2134898 torch/distributed/run.py:792] 
0: W0902 18:58:55.639000 2134898 torch/distributed/run.py:792] *****************************************
0: W0902 18:58:55.639000 2134898 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
0: W0902 18:58:55.639000 2134898 torch/distributed/run.py:792] *****************************************
1: W0902 18:58:55.819000 3856158 torch/distributed/run.py:792] 
1: W0902 18:58:55.819000 3856158 torch/distributed/run.py:792] *****************************************
1: W0902 18:58:55.819000 3856158 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
1: W0902 18:58:55.819000 3856158 torch/distributed/run.py:792] *****************************************
2: W0902 18:58:56.452000 3393108 torch/distributed/run.py:792] 
2: W0902 18:58:56.452000 3393108 torch/distributed/run.py:792] *****************************************
2: W0902 18:58:56.452000 3393108 torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
2: W0902 18:58:56.452000 3393108 torch/distributed/run.py:792] *****************************************
3: [2025-09-02 18:59:20,646] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:284556] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`
3: [2025-09-02 18:59:20,646] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:284556] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing
1: [2025-09-02 18:59:21,727] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:3856235] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`
1: [2025-09-02 18:59:21,728] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:3856235] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing
0: [2025-09-02 18:59:23,380] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:2134977] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`
0: [2025-09-02 18:59:23,380] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:2134977] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing
2: [2025-09-02 18:59:23,899] [INFO] [axolotl.utils.schemas.validation.check_eval_packing:119] [PID:3393184] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`
2: [2025-09-02 18:59:23,899] [INFO] [axolotl.utils.schemas.validation.hint_sample_packing_padding:218] [PID:3393184] [RANK:0] Setting `pad_to_sequence_len: true` to prevent memory leaks when sample_packing
3: [2025-09-02 18:59:25,238] [INFO] [axolotl.utils.data.sft._load_raw_datasets:314] [PID:284557] [RANK:1] Loading raw datasets...
3: [2025-09-02 18:59:25,510] [INFO] [axolotl.utils.data.wrappers.get_dataset_wrapper:88] [PID:284557] [RANK:1] Loading dataset: /lustre/fswork/projects/rech/qwv/udv55np/dataset/math/hf/no_thinking_text/generator/default-d32b2cae8ea7e541/0.0.0 with base_type: chat_template and prompt_style: None
0: [2025-09-02 18:59:27,157] [INFO] [axolotl.cli.config.load_cfg:245] [PID:2134977] [RANK:0] config:
0: {
0:   "activation_offloading": false,
0:   "auto_resume_from_checkpoints": true,
0:   "axolotl_config_path": "/lustre/fswork/projects/rech/dgo/udv55np/train/tmp/1756826506262014665.yaml",
0:   "base_model": "/lustre/fswork/projects/rech/qwv/udv55np/Qwen/Qwen2.5-14B_ift",
0:   "base_model_config": "/lustre/fswork/projects/rech/qwv/udv55np/Qwen/Qwen2.5-14B_ift",
0:   "batch_size": 16,
0:   "bf16": true,
0:   "capabilities": {
0:     "bf16": true,
0:     "compute_capability": "sm_90",
0:     "fp8": false,
0:     "n_gpu": 16,
0:     "n_node": 1
0:   },
0:   "chat_template": "qwen_25",
0:   "context_parallel_size": 1,
0:   "dataloader_num_workers": 16,
0:   "dataloader_pin_memory": true,
0:   "dataloader_prefetch_factor": 256,
0:   "dataset_prepared_path": "/lustre/fsn1/projects/rech/dgo/udv55np/dataset_math/Qwen3-235B-A22B/0",
0:   "dataset_processes": 192,
0:   "datasets": [
0:     {
0:       "chat_template": "tokenizer_default",
0:       "field_messages": "conversations",
0:       "message_property_mappings": {
0:         "content": "content",
0:         "role": "role"
0:       },
0:       "path": "/lustre/fswork/projects/rech/qwv/udv55np/dataset/math/hf/no_thinking_text/generator/default-d32b2cae8ea7e541/0.0.0",
0:       "trust_remote_code": false,
0:       "type": "chat_template"
0:     }
0:   ],
0:   "ddp": true,
0:   "deepspeed": {
0:     "bf16": {
0:       "enabled": true
0:     },
0:     "gradient_accumulation_steps": "auto",
0:     "gradient_clipping": "auto",
0:     "train_batch_size": "auto",
0:     "train_micro_batch_size_per_gpu": "auto",
0:     "wall_clock_breakdown": false,
0:     "zero_optimization": {
0:       "contiguous_gradients": true,
0:       "overlap_comm": true,
0:       "reduce_bucket_size": "auto",
0:       "stage": 3,
0:       "stage3_gather_16bit_weights_on_model_save": true,
0:       "stage3_param_persistence_threshold": "auto",
0:       "stage3_prefetch_bucket_size": "auto",
0:       "sub_group_size": 0
0:     }
0:   },
0:   "device": "cuda:0",
0:   "device_map": {
0:     "": 0
0:   },
0:   "dion_rank_fraction": 1.0,
0:   "dion_rank_multiple_of": 1,
0:   "env_capabilities": {
0:     "torch_version": "2.6.0"
0:   },
0:   "eval_batch_size": 1,
0:   "eval_causal_lm_metrics": [
0:     "sacrebleu",
0:     "comet",
0:     "ter",
0:     "chrf"
0:   ],
0:   "eval_max_new_tokens": 128,
0:   "eval_sample_packing": true,
0:   "eval_table_size": 0,
0:   "evals_per_epoch": 0,
0:   "flash_attention": true,
0:   "fp16": false,
0:   "gradient_accumulation_steps": 1,
0:   "gradient_checkpointing": true,
0:   "gradient_checkpointing_kwargs": {
0:     "use_reentrant": true
0:   },
0:   "learning_rate": 1e-06,
0:   "lisa_layers_attribute": "model.layers",
0:   "load_best_model_at_end": false,
0:   "load_in_4bit": false,
0:   "load_in_8bit": false,
0:   "local_rank": 0,
0:   "logging_steps": 10,
0:   "lora_dropout": 0.0,
0:   "loraplus_lr_embedding": 1e-06,
0:   "lr_scheduler": "warmup_stable_decay",
0:   "lr_scheduler_kwargs": {
0:     "min_lr_ratio": 0.1,
0:     "num_decay_steps": 300
0:   },
0:   "max_prompt_len": 512,
0:   "mean_resizing_embeddings": false,
0:   "micro_batch_size": 1,
0:   "model_config_type": "qwen2",
0:   "num_epochs": 1.0,
0:   "optimizer": "adamw_torch_fused",
0:   "output_dir": "/lustre/fswork/projects/rech/dgo/udv55np/math/Qwen3-235B-A22B/Qwen2.5-14B_ift/0",
0:   "pad_to_sequence_len": true,
0:   "pretrain_multipack_attn": true,
0:   "pretrain_multipack_buffer_size": 10000,
0:   "profiler_steps_start": 0,
0:   "qlora_sharded_model_loading": false,
0:   "ray_num_workers": 1,
0:   "resources_per_worker": {
0:     "GPU": 1
0:   },
0:   "sample_packing": true,
0:   "sample_packing_bin_size": 200,
0:   "sample_packing_group_size": 100000,
0:   "save_only_model": false,
0:   "save_safetensors": true,
0:   "save_steps": 0.2,
0:   "save_total_limit": 20,
0:   "sequence_len": 16384,
0:   "shuffle_before_merging_datasets": false,
0:   "shuffle_merged_datasets": true,
0:   "skip_prepare_dataset": false,
0:   "special_tokens": {
0:     "bos_token": "<|im_start|>",
0:     "eos_token": "<|im_end|>",
0:     "pad_token": "<|endoftext|>"
0:   },
0:   "strict": false,
0:   "tensor_parallel_size": 1,
0:   "tf32": false,
0:   "tiled_mlp_use_original_mlp": true,
0:   "tokenizer_config": "/lustre/fswork/projects/rech/qwv/udv55np/Qwen/Qwen2.5-14B_ift",
0:   "torch_dtype": "torch.bfloat16",
0:   "train_on_inputs": false,
0:   "trl": {
0:     "log_completions": false,
0:     "mask_truncated_completions": false,
0:     "ref_model_mixup_alpha": 0.9,
0:     "ref_model_sync_steps": 64,
0:     "scale_rewards": true,
0:     "sync_ref_model": false,
0:     "use_vllm": false,
0:     "vllm_server_host": "0.0.0.0",
0:     "vllm_server_port": 8000
0:   },
0:   "use_ray": false,
0:   "use_tensorboard": true,
0:   "val_set_size": 0.0,
0:   "vllm": {
0:     "device": "auto",
0:     "dtype": "auto",
0:     "gpu_memory_utilization": 0.9,
0:     "host": "0.0.0.0",
0:     "port": 8000
0:   },
0:   "warmup_steps": 150,
0:   "weight_decay": 0.0,
0:   "world_size": 16
0: }
0: [2025-09-02 18:59:27,159] [INFO] [axolotl.cli.checks.check_user_token:35] [PID:2134977] [RANK:0] Skipping HuggingFace token verification because HF_HUB_OFFLINE is set to True. Only local files will be used.
3: 
Tokenizing Prompts (num_proc=192):   0%|          | 0/321773 [00:00<?, ? examples/s]
Tokenizing Prompts (num_proc=192):   0%|          | 1000/321773 [00:07<41:34, 128.60 examples/s]
Tokenizing Prompts (num_proc=192):   1%|          | 2000/321773 [00:08<18:38, 285.86 examples/s]
Tokenizing Prompts (num_proc=192):   1%|          | 3000/321773 [00:08<10:21, 513.28 examples/s]
Tokenizing Prompts (num_proc=192):   2%|▏         | 5000/321773 [00:08<04:37, 1143.09 examples/s]
Tokenizing Prompts (num_proc=192):   2%|▏         | 7000/321773 [00:09<03:16, 1603.57 examples/s]
Tokenizing Prompts (num_proc=192):   3%|▎         | 9000/321773 [00:09<02:05, 2491.44 examples/s]
Tokenizing Prompts (num_proc=192):   3%|▎         | 11000/321773 [00:09<01:38, 3143.52 examples/s]
Tokenizing Prompts (num_proc=192):   4%|▎         | 12000/321773 [00:09<01:32, 3331.12 examples/s]
Tokenizing Prompts (num_proc=192):   4%|▍         | 13000/321773 [00:09<01:19, 3876.87 examples/s]
Tokenizing Prompts (num_proc=192):   5%|▍ 
3:         | 16000/321773 [00:10<00:45, 6718.05 examples/s]
Tokenizing Prompts (num_proc=192):   6%|▌         | 19000/321773 [00:10<00:33, 9114.77 examples/s]
Tokenizing Prompts (num_proc=192):   7%|▋         | 23000/321773 [00:10<00:22, 13402.62 examples/s]
Tokenizing Prompts (num_proc=192):   8%|▊         | 26000/321773 [00:10<00:25, 11522.97 examples/s]
Tokenizing Prompts (num_proc=192):   9%|▊         | 28000/321773 [00:11<00:35, 8265.75 examples/s] 
Tokenizing Prompts (num_proc=192):   9%|▉         | 30000/321773 [00:11<00:32, 9052.47 examples/s]
Tokenizing Prompts (num_proc=192):  10%|▉         | 32000/321773 [00:11<00:36, 8035.37 examples/s]
Tokenizing Prompts (num_proc=192):  11%|█▏        | 37000/321773 [00:11<00:22, 12484.18 examples/s]
Tokenizing Prompts (num_proc=192):  12%|█▏        | 39000/321773 [00:11<00:22, 12537.61 examples/s]
Tokenizing Prompts (num_proc=192):  13%|█▎        | 41000/321773 [00:12<00:22, 12552.09 examples/s]
Tokenizing Prompts (num_proc=192):  14%|█▎
3:         | 43676/321773 [00:12<00:20, 13559.72 examples/s]
Tokenizing Prompts (num_proc=192):  14%|█▍        | 45676/321773 [00:12<00:37, 7369.49 examples/s] 
Tokenizing Prompts (num_proc=192):  15%|█▌        | 48676/321773 [00:13<00:34, 7925.22 examples/s]
Tokenizing Prompts (num_proc=192):  16%|█▋        | 52352/321773 [00:13<00:25, 10493.94 examples/s]
Tokenizing Prompts (num_proc=192):  17%|█▋        | 54352/321773 [00:13<00:24, 10924.24 examples/s]
Tokenizing Prompts (num_proc=192):  18%|█▊        | 56352/321773 [00:13<00:28, 9289.49 examples/s] 
Tokenizing Prompts (num_proc=192):  18%|█▊        | 58028/321773 [00:14<00:30, 8695.81 examples/s]
Tokenizing Prompts (num_proc=192):  19%|█▊        | 59704/321773 [00:14<00:27, 9699.29 examples/s]
Tokenizing Prompts (num_proc=192):  19%|█▉        | 61056/321773 [00:14<00:26, 9991.60 examples/s]
Tokenizing Prompts (num_proc=192):  20%|█▉        | 63056/321773 [00:14<00:31, 8158.52 examples/s]
Tokenizing Prompts (num_proc=192): 
3:  20%|██        | 65056/321773 [00:14<00:35, 7261.23 examples/s]
Tokenizing Prompts (num_proc=192):  21%|██        | 67732/321773 [00:15<00:30, 8367.22 examples/s]
Tokenizing Prompts (num_proc=192):  22%|██▏       | 71084/321773 [00:15<00:21, 11697.63 examples/s]
Tokenizing Prompts (num_proc=192):  23%|██▎       | 72760/321773 [00:15<00:20, 12142.59 examples/s]
Tokenizing Prompts (num_proc=192):  23%|██▎       | 75436/321773 [00:15<00:21, 11235.07 examples/s]
Tokenizing Prompts (num_proc=192):  24%|██▍       | 77436/321773 [00:15<00:21, 11479.77 examples/s]
Tokenizing Prompts (num_proc=192):  24%|██▍       | 78788/321773 [00:16<00:22, 10774.12 examples/s]
Tokenizing Prompts (num_proc=192):  25%|██▌       | 81140/321773 [00:16<00:20, 11729.30 examples/s]
Tokenizing Prompts (num_proc=192):  26%|██▌       | 83816/321773 [00:16<00:18, 12957.51 examples/s]
Tokenizing Prompts (num_proc=192):  27%|██▋       | 86492/321773 [00:16<00:17, 13785.25 examples/s]
Token
3: izing Prompts (num_proc=192):  27%|██▋       | 88168/321773 [00:16<00:18, 12898.81 examples/s]
Tokenizing Prompts (num_proc=192):  28%|██▊       | 89844/321773 [00:17<00:24, 9369.35 examples/s] 
Tokenizing Prompts (num_proc=192):  30%|██▉       | 95548/321773 [00:17<00:14, 15619.35 examples/s]
Tokenizing Prompts (num_proc=192):  31%|███       | 99576/321773 [00:17<00:12, 17957.30 examples/s]
Tokenizing Prompts (num_proc=192):  32%|███▏      | 102252/321773 [00:17<00:12, 17425.65 examples/s]
Tokenizing Prompts (num_proc=192):  33%|███▎      | 104604/321773 [00:17<00:14, 14836.35 examples/s]
Tokenizing Prompts (num_proc=192):  33%|███▎      | 106632/321773 [00:17<00:15, 13804.67 examples/s]
Tokenizing Prompts (num_proc=192):  34%|███▍      | 109308/321773 [00:18<00:13, 16051.27 examples/s]
Tokenizing Prompts (num_proc=192):  35%|███▍      | 111660/321773 [00:18<00:18, 11633.51 examples/s]
Tokenizing Prompts (num_proc=192):  35%|███▌      | 
3: 113660/321773 [00:18<00:17, 11793.91 examples/s]
Tokenizing Prompts (num_proc=192):  36%|███▌      | 115660/321773 [00:19<00:26, 7858.69 examples/s] 
Tokenizing Prompts (num_proc=192):  36%|███▋      | 117336/321773 [00:19<00:24, 8183.85 examples/s]
Tokenizing Prompts (num_proc=192):  38%|███▊      | 122040/321773 [00:19<00:22, 8757.69 examples/s]
Tokenizing Prompts (num_proc=192):  39%|███▊      | 124392/321773 [00:19<00:20, 9764.14 examples/s]
Tokenizing Prompts (num_proc=192):  39%|███▉      | 126068/321773 [00:20<00:19, 9786.40 examples/s]
Tokenizing Prompts (num_proc=192):  40%|███▉      | 127420/321773 [00:20<00:20, 9373.78 examples/s]
Tokenizing Prompts (num_proc=192):  40%|████      | 128772/321773 [00:20<00:20, 9304.94 examples/s]
Tokenizing Prompts (num_proc=192):  41%|████      | 130448/321773 [00:20<00:20, 9437.12 examples/s]
Tokenizing Prompts (num_proc=192):  41%|████      | 132124/321773 [00:20<00:19, 9568.90 examples/s]
To
3: kenizing Prompts (num_proc=192):  42%|████▏     | 134476/321773 [00:20<00:17, 10747.19 examples/s]
Tokenizing Prompts (num_proc=192):  43%|████▎     | 137856/321773 [00:21<00:15, 11981.99 examples/s]
Tokenizing Prompts (num_proc=192):  43%|████▎     | 139208/321773 [00:21<00:18, 9673.00 examples/s] 
Tokenizing Prompts (num_proc=192):  46%|████▌     | 146912/321773 [00:21<00:10, 17267.28 examples/s]
Tokenizing Prompts (num_proc=192):  46%|████▋     | 149588/321773 [00:21<00:11, 14959.04 examples/s]
Tokenizing Prompts (num_proc=192):  48%|████▊     | 153616/321773 [00:22<00:10, 15773.59 examples/s]
Tokenizing Prompts (num_proc=192):  48%|████▊     | 155616/321773 [00:22<00:12, 13297.63 examples/s]
Tokenizing Prompts (num_proc=192):  49%|████▉     | 158968/321773 [00:22<00:12, 13243.22 examples/s]
Tokenizing Prompts (num_proc=192):  50%|█████     | 160996/321773 [00:22<00:13, 11915.14 examples/s]
Tokenizing Prompts (num_pro
3: c=192):  51%|█████     | 163996/321773 [00:23<00:13, 12083.29 examples/s]
Tokenizing Prompts (num_proc=192):  51%|█████▏    | 165348/321773 [00:23<00:15, 9800.87 examples/s] 
Tokenizing Prompts (num_proc=192):  52%|█████▏    | 168700/321773 [00:23<00:13, 11429.76 examples/s]
Tokenizing Prompts (num_proc=192):  53%|█████▎    | 171700/321773 [00:23<00:11, 12975.64 examples/s]
Tokenizing Prompts (num_proc=192):  54%|█████▍    | 173376/321773 [00:23<00:12, 12174.84 examples/s]
Tokenizing Prompts (num_proc=192):  55%|█████▍    | 176728/321773 [00:24<00:11, 12855.59 examples/s]
Tokenizing Prompts (num_proc=192):  56%|█████▌    | 179080/321773 [00:24<00:11, 12412.05 examples/s]
Tokenizing Prompts (num_proc=192):  57%|█████▋    | 182080/321773 [00:24<00:09, 14042.85 examples/s]
Tokenizing Prompts (num_proc=192):  57%|█████▋    | 184756/321773 [00:24<00:10, 13372.71 examples/s]
Tokenizing Prompts (num_proc=192):  
3: 59%|█████▉    | 189756/321773 [00:24<00:07, 17876.22 examples/s]
Tokenizing Prompts (num_proc=192):  60%|█████▉    | 192108/321773 [00:25<00:07, 17095.18 examples/s]
Tokenizing Prompts (num_proc=192):  60%|██████    | 194460/321773 [00:25<00:08, 14323.26 examples/s]
Tokenizing Prompts (num_proc=192):  62%|██████▏   | 198136/321773 [00:25<00:08, 15290.31 examples/s]
Tokenizing Prompts (num_proc=192):  62%|██████▏   | 200488/321773 [00:25<00:09, 13028.90 examples/s]
Tokenizing Prompts (num_proc=192):  63%|██████▎   | 202164/321773 [00:25<00:09, 11997.37 examples/s]
Tokenizing Prompts (num_proc=192):  63%|██████▎   | 203516/321773 [00:26<00:10, 11727.61 examples/s]
Tokenizing Prompts (num_proc=192):  64%|██████▎   | 204868/321773 [00:26<00:09, 12007.68 examples/s]
Tokenizing Prompts (num_proc=192):  64%|██████▍   | 206544/321773 [00:26<00:15, 7246.10 examples/s] 
Tokenizing Prompts (num_proc=19
3: 2):  65%|██████▌   | 209896/321773 [00:26<00:10, 10740.11 examples/s]
Tokenizing Prompts (num_proc=192):  66%|██████▌   | 211896/321773 [00:26<00:10, 10950.05 examples/s]
Tokenizing Prompts (num_proc=192):  66%|██████▋   | 213896/321773 [00:27<00:09, 11728.33 examples/s]
Tokenizing Prompts (num_proc=192):  67%|██████▋   | 216924/321773 [00:27<00:07, 13519.31 examples/s]
Tokenizing Prompts (num_proc=192):  68%|██████▊   | 219600/321773 [00:27<00:07, 14493.64 examples/s]
Tokenizing Prompts (num_proc=192):  69%|██████▉   | 221276/321773 [00:27<00:07, 13794.71 examples/s]
Tokenizing Prompts (num_proc=192):  69%|██████▉   | 223276/321773 [00:27<00:06, 14092.46 examples/s]
Tokenizing Prompts (num_proc=192):  70%|██████▉   | 224952/321773 [00:27<00:07, 13002.96 examples/s]
Tokenizing Prompts (num_proc=192):  70%|███████   | 226628/321773 [00:27<00:07, 12991.74 examples/s]
Tokenizing Prompts (
3: num_proc=192):  72%|███████▏  | 231332/321773 [00:28<00:04, 19985.53 examples/s]
Tokenizing Prompts (num_proc=192):  73%|███████▎  | 234360/321773 [00:28<00:04, 21578.81 examples/s]
Tokenizing Prompts (num_proc=192):  74%|███████▎  | 237036/321773 [00:28<00:05, 16924.96 examples/s]
Tokenizing Prompts (num_proc=192):  75%|███████▌  | 241388/321773 [00:28<00:03, 21813.09 examples/s]
Tokenizing Prompts (num_proc=192):  76%|███████▌  | 245064/321773 [00:28<00:03, 23930.01 examples/s]
Tokenizing Prompts (num_proc=192):  77%|███████▋  | 247768/321773 [00:29<00:06, 11150.30 examples/s]
Tokenizing Prompts (num_proc=192):  78%|███████▊  | 250472/321773 [00:29<00:06, 11703.18 examples/s]
Tokenizing Prompts (num_proc=192):  79%|███████▊  | 252824/321773 [00:29<00:07, 9437.76 examples/s] 
Tokenizing Prompts (num_proc=192):  79%|███████▉  | 255500/321773 [00:29<00:05, 11389.15 exa
3: mples/s]
Tokenizing Prompts (num_proc=192):  80%|████████  | 258528/321773 [00:30<00:04, 13544.75 examples/s]
Tokenizing Prompts (num_proc=192):  81%|████████  | 260880/321773 [00:30<00:04, 13404.34 examples/s]
Tokenizing Prompts (num_proc=192):  82%|████████▏ | 262908/321773 [00:30<00:04, 12458.45 examples/s]
Tokenizing Prompts (num_proc=192):  82%|████████▏ | 264584/321773 [00:30<00:04, 12611.36 examples/s]
Tokenizing Prompts (num_proc=192):  83%|████████▎ | 266612/321773 [00:30<00:05, 11030.94 examples/s]
Tokenizing Prompts (num_proc=192):  83%|████████▎ | 268288/321773 [00:31<00:04, 10785.30 examples/s]
Tokenizing Prompts (num_proc=192):  85%|████████▍ | 272640/321773 [00:31<00:02, 16783.18 examples/s]
Tokenizing Prompts (num_proc=192):  86%|████████▌ | 275640/321773 [00:31<00:03, 13232.34 examples/s]
Tokenizing Prompts (num_proc=192):  86%|████████▋
3:  | 278316/321773 [00:31<00:03, 13648.03 examples/s]
Tokenizing Prompts (num_proc=192):  87%|████████▋ | 280668/321773 [00:31<00:03, 10898.83 examples/s]
Tokenizing Prompts (num_proc=192):  88%|████████▊ | 282696/321773 [00:32<00:03, 10797.91 examples/s]
Tokenizing Prompts (num_proc=192):  89%|████████▊ | 285372/321773 [00:32<00:03, 12049.84 examples/s]
Tokenizing Prompts (num_proc=192):  89%|████████▉ | 287372/321773 [00:32<00:02, 12532.53 examples/s]
Tokenizing Prompts (num_proc=192):  90%|████████▉ | 289048/321773 [00:32<00:02, 12836.05 examples/s]
Tokenizing Prompts (num_proc=192):  91%|█████████▏| 293752/321773 [00:32<00:01, 17684.96 examples/s]
Tokenizing Prompts (num_proc=192):  92%|█████████▏| 297456/321773 [00:32<00:01, 19206.11 examples/s]
Tokenizing Prompts (num_proc=192):  93%|█████████▎| 299484/321773 [00:33<00:01, 13543.34 examples/s]
Tokenizing Pro
3: mpts (num_proc=192):  94%|█████████▎| 301512/321773 [00:33<00:01, 13618.95 examples/s]
Tokenizing Prompts (num_proc=192):  94%|█████████▍| 303540/321773 [00:33<00:01, 10297.11 examples/s]
Tokenizing Prompts (num_proc=192):  95%|█████████▍| 304892/321773 [00:34<00:02, 6584.67 examples/s] 
Tokenizing Prompts (num_proc=192):  95%|█████████▌| 306920/321773 [00:34<00:02, 5077.01 examples/s]
Tokenizing Prompts (num_proc=192):  96%|█████████▌| 308270/321773 [00:35<00:03, 4363.32 examples/s]
Tokenizing Prompts (num_proc=192):  96%|█████████▌| 309620/321773 [00:35<00:02, 4829.96 examples/s]
Tokenizing Prompts (num_proc=192):  97%|█████████▋| 310970/321773 [00:35<00:01, 5517.45 examples/s]
Tokenizing Prompts (num_proc=192):  97%|█████████▋| 312997/321773 [00:35<00:01, 7286.19 examples/s]
Tokenizing Prompts (num_proc=192):  98%|█████████▊| 314
3: 348/321773 [00:36<00:01, 5484.63 examples/s]
Tokenizing Prompts (num_proc=192):  98%|█████████▊| 315698/321773 [00:36<00:01, 4839.05 examples/s]
Tokenizing Prompts (num_proc=192):  99%|█████████▊| 317048/321773 [00:36<00:01, 4394.45 examples/s]
Tokenizing Prompts (num_proc=192):  99%|█████████▉| 318398/321773 [00:37<00:00, 5390.08 examples/s]
Tokenizing Prompts (num_proc=192): 100%|█████████▉| 320423/321773 [00:37<00:00, 7179.95 examples/s]
Tokenizing Prompts (num_proc=192): 100%|██████████| 321773/321773 [00:37<00:00, 5973.17 examples/s]
Tokenizing Prompts (num_proc=192): 100%|██████████| 321773/321773 [00:37<00:00, 8478.51 examples/s]
3: 
Dropping Long Sequences (>16384) (num_proc=192):   0%|          | 0/321773 [00:00<?, ? examples/s]
Dropping Long Sequences (>16384) (num_proc=192):   0%|          | 1000/321773 [00:01<09:44, 548.90 examples/s]
Dropping Long Sequences (>16384) (num_proc=192):   5%|▍         | 15000/321773 [00:01<00:28, 10641.73 examples/s]
Dropping Long Sequences (>16384) (num_proc=192):   8%|▊         | 24676/321773 [00:02<00:18, 16454.79 examples/s]
Dropping Long Sequences (>16384) (num_proc=192):  10%|▉         | 30732/321773 [00:02<00:15, 18680.29 examples/s]
Dropping Long Sequences (>16384) (num_proc=192):  11%|█         | 36112/321773 [00:02<00:13, 21058.71 examples/s]
Dropping Long Sequences (>16384) (num_proc=192):  13%|█▎        | 40492/321773 [00:02<00:14, 18996.48 examples/s]
Dropping Long Sequences (>16384) (num_proc=192):  14%|█▎        | 43844/321773 [00:03<00:14, 19470.16 examples/s]
Dropping Long Sequences (>16384) (num_proc=192):  15%|█▌        | 48548/321773 [00:03<00:11, 23185.99 example
3: s/s]
Dropping Long Sequences (>16384) (num_proc=192):  16%|█▌        | 52224/321773 [00:03<00:10, 25109.15 examples/s]
Dropping Long Sequences (>16384) (num_proc=192):  54%|█████▍    | 173756/321773 [00:03<00:00, 257188.15 examples/s]
Dropping Long Sequences (>16384) (num_proc=192):  66%|██████▋   | 213896/321773 [00:03<00:00, 260962.67 examples/s]
Dropping Long Sequences (>16384) (num_proc=192):  78%|███████▊  | 249796/321773 [00:03<00:00, 225600.38 examples/s]
Dropping Long Sequences (>16384) (num_proc=192):  87%|████████▋ | 280020/321773 [00:03<00:00, 219928.40 examples/s]
Dropping Long Sequences (>16384) (num_proc=192):  96%|█████████▌| 307593/321773 [00:04<00:00, 177091.06 examples/s]
Dropping Long Sequences (>16384) (num_proc=192): 100%|██████████| 321773/321773 [00:04<00:00, 68433.68 examples/s] 
3: 
Drop Samples with Zero Trainable Tokens (num_proc=192):   0%|          | 0/315947 [00:00<?, ? examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):   0%|          | 1000/315947 [00:02<10:32, 497.84 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):   6%|▌         | 19000/315947 [00:02<00:23, 12405.63 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):  10%|▉         | 30230/315947 [00:02<00:15, 18618.36 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):  12%|█▏        | 37982/315947 [00:02<00:14, 18827.53 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):  14%|█▍        | 43796/315947 [00:03<00:13, 19456.77 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):  15%|█▌        | 48442/315947 [00:03<00:12, 22066.58 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):  21%|██▏       | 67734/315947 [00:03<00:05, 42162.92 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=1
3: 92):  24%|██▍       | 75610/315947 [00:03<00:05, 43374.10 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):  26%|██▌       | 82548/315947 [00:03<00:05, 41894.15 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):  28%|██▊       | 88486/315947 [00:03<00:05, 41648.69 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):  30%|██▉       | 94070/315947 [00:03<00:05, 39887.99 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):  31%|███▏      | 99362/315947 [00:04<00:05, 40725.77 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):  33%|███▎      | 103946/315947 [00:04<00:05, 38635.30 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):  34%|███▍      | 108884/315947 [00:04<00:05, 40702.57 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):  40%|███▉      | 125344/315947 [00:04<00:02, 69558.66 examples/s]
Drop Samples with Zero Trainable Tokens (num_pro
3: c=192):  80%|███████▉  | 252052/315947 [00:04<00:00, 369109.63 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192):  94%|█████████▎| 296112/315947 [00:04<00:00, 228684.13 examples/s]
Drop Samples with Zero Trainable Tokens (num_proc=192): 100%|██████████| 315947/315947 [00:05<00:00, 57249.08 examples/s] 
3: 
Add position_id column (Sample Packing) (num_proc=192):   0%|          | 0/315947 [00:00<?, ? examples/s]
Add position_id column (Sample Packing) (num_proc=192):   0%|          | 1000/315947 [00:02<11:19, 463.69 examples/s]
Add position_id column (Sample Packing) (num_proc=192):   3%|▎         | 10000/315947 [00:02<00:51, 5990.50 examples/s]
Add position_id column (Sample Packing) (num_proc=192):   7%|▋         | 23000/315947 [00:02<00:18, 15986.23 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  11%|█         | 34000/315947 [00:02<00:11, 25599.61 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  14%|█▍        | 45000/315947 [00:02<00:07, 36215.98 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  17%|█▋        | 55000/315947 [00:02<00:05, 45761.18 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  21%|██        | 67000/315947 [00:02<00:04, 58922.50 examples/s]
Add position_id column (Sample Packing) (num_proc=192): 
3:  25%|██▍       | 77938/315947 [00:02<00:04, 58589.19 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  28%|██▊       | 87106/315947 [00:03<00:05, 42119.45 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  40%|████      | 127362/315947 [00:03<00:01, 94557.94 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  45%|████▌     | 142530/315947 [00:03<00:01, 96545.75 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  50%|████▉     | 156406/315947 [00:03<00:01, 104362.52 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  54%|█████▍    | 170220/315947 [00:03<00:01, 106994.69 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  61%|██████    | 192910/315947 [00:03<00:00, 128383.41 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  66%|██████▌   | 207721/315947 [00:04<00:01, 99321.02 examples/s] 
Add position_id column (
3: Sample Packing) (num_proc=192):  72%|███████▏  | 226407/315947 [00:04<00:00, 116855.65 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  77%|███████▋  | 243924/315947 [00:04<00:00, 127103.90 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  83%|████████▎ | 261891/315947 [00:04<00:00, 138776.78 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  88%|████████▊ | 278116/315947 [00:04<00:00, 137691.98 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  93%|█████████▎| 293372/315947 [00:04<00:00, 111443.49 examples/s]
Add position_id column (Sample Packing) (num_proc=192):  97%|█████████▋| 306272/315947 [00:05<00:00, 97509.17 examples/s] 
Add position_id column (Sample Packing) (num_proc=192): 100%|██████████| 315947/315947 [00:05<00:00, 56185.04 examples/s]
3: 
Saving the dataset (0/192 shards):   0%|          | 0/315947 [00:00<?, ? examples/s]
Saving the dataset (0/192 shards):   1%|          | 1646/315947 [00:01<05:52, 892.52 examples/s]
Saving the dataset (1/192 shards):   1%|          | 1646/315947 [00:01<05:52, 892.52 examples/s]
Saving the dataset (2/192 shards):   1%|          | 3292/315947 [00:01<05:50, 892.52 examples/s]
Saving the dataset (3/192 shards):   2%|▏         | 4938/315947 [00:01<05:48, 892.52 examples/s]
Saving the dataset (4/192 shards):   3%|▎         | 8230/315947 [00:01<05:44, 892.52 examples/s]
Saving the dataset (5/192 shards):   3%|▎         | 8230/315947 [00:01<05:44, 892.52 examples/s]
Saving the dataset (6/192 shards):   3%|▎         | 9876/315947 [00:01<05:42, 892.52 examples/s]
Saving the dataset (7/192 shards):   4%|▎         | 11522/315947 [00:01<05:41, 892.52 examples/s]
Saving the dataset (8/192 shards):   4%|▍         | 13168/315947 [00:01<05:39, 892.52 examples/s]
Saving the dataset (9/192 shards):   5%|▍        
3:  | 14814/315947 [00:01<05:37, 892.52 examples/s]
Saving the dataset (10/192 shards):   5%|▌         | 16460/315947 [00:01<05:35, 892.52 examples/s]
Saving the dataset (11/192 shards):   6%|▌         | 18106/315947 [00:01<05:33, 892.52 examples/s]
Saving the dataset (12/192 shards):   7%|▋         | 21398/315947 [00:01<05:30, 892.52 examples/s]
Saving the dataset (13/192 shards):   7%|▋         | 21398/315947 [00:01<05:30, 892.52 examples/s]
Saving the dataset (14/192 shards):   7%|▋         | 23044/315947 [00:01<05:28, 892.52 examples/s]
Saving the dataset (15/192 shards):   8%|▊         | 24690/315947 [00:01<05:26, 892.52 examples/s]
Saving the dataset (16/192 shards):   8%|▊         | 26336/315947 [00:01<05:24, 892.52 examples/s]
Saving the dataset (17/192 shards):   9%|▉         | 29628/315947 [00:01<05:20, 892.52 examples/s]
Saving the dataset (18/192 shards):   9%|▉         | 29628/315947 [00:01<05:20, 892.52 examples/s]
Saving the dataset (19/192 shards):  10%|▉         | 31274/31594
3: 7 [00:01<05:18, 892.52 examples/s]
Saving the dataset (20/192 shards):  10%|█         | 32920/315947 [00:01<05:17, 892.52 examples/s]
Saving the dataset (21/192 shards):  11%|█▏        | 36212/315947 [00:01<05:13, 892.52 examples/s]
Saving the dataset (22/192 shards):  11%|█▏        | 36212/315947 [00:01<05:13, 892.52 examples/s]
Saving the dataset (23/192 shards):  12%|█▏        | 37858/315947 [00:01<05:11, 892.52 examples/s]
Saving the dataset (24/192 shards):  13%|█▎        | 41150/315947 [00:01<05:07, 892.52 examples/s]
Saving the dataset (25/192 shards):  13%|█▎        | 41150/315947 [00:01<05:07, 892.52 examples/s]
Saving the dataset (26/192 shards):  14%|█▎        | 42796/315947 [00:01<05:06, 892.52 examples/s]
Saving the dataset (27/192 shards):  15%|█▍        | 46088/315947 [00:01<05:02, 892.52 examples/s]
Saving the dataset (28/192 shards):  15%|█▍        | 46088/315947 [00:01<05:02, 892.52 examples/s]
Saving the dataset (29/192 shards):  15%|█▌        | 47734/3
3: 15947 [00:01<05:00, 892.52 examples/s]
Saving the dataset (30/192 shards):  16%|█▌        | 49380/315947 [00:01<04:58, 892.52 examples/s]
Saving the dataset (31/192 shards):  16%|█▌        | 51026/315947 [00:01<04:56, 892.52 examples/s]
Saving the dataset (32/192 shards):  17%|█▋        | 52672/315947 [00:01<04:54, 892.52 examples/s]
Saving the dataset (33/192 shards):  17%|█▋        | 54318/315947 [00:01<04:53, 892.52 examples/s]
Saving the dataset (34/192 shards):  19%|█▉        | 59256/315947 [00:01<04:47, 892.52 examples/s]
Saving the dataset (35/192 shards):  19%|█▉        | 59256/315947 [00:01<04:47, 892.52 examples/s]
Saving the dataset (36/192 shards):  19%|█▉        | 59256/315947 [00:01<04:47, 892.52 examples/s]
Saving the dataset (37/192 shards):  20%|█▉        | 62548/315947 [00:01<04:43, 892.52 examples/s]
Saving the dataset (38/192 shards):  20%|█▉        | 62548/315947 [00:01<04:43, 892.52 examples/s]
Saving the dataset (39/192 shards):  20%|██        | 6
3: 4194/315947 [00:01<04:42, 892.52 examples/s]
Saving the dataset (40/192 shards):  21%|██        | 65840/315947 [00:01<04:40, 892.52 examples/s]
Saving the dataset (41/192 shards):  21%|██▏       | 67486/315947 [00:01<04:38, 892.52 examples/s]
Saving the dataset (42/192 shards):  22%|██▏       | 69132/315947 [00:01<04:36, 892.52 examples/s]
Saving the dataset (43/192 shards):  23%|██▎       | 72424/315947 [00:01<04:32, 892.52 examples/s]
Saving the dataset (44/192 shards):  23%|██▎       | 72424/315947 [00:01<04:32, 892.52 examples/s]
Saving the dataset (45/192 shards):  23%|██▎       | 74070/315947 [00:01<04:31, 892.52 examples/s]
Saving the dataset (46/192 shards):  24%|██▍       | 77362/315947 [00:01<04:27, 892.52 examples/s]
Saving the dataset (47/192 shards):  24%|██▍       | 77362/315947 [00:01<04:27, 892.52 examples/s]
Saving the dataset (48/192 shards):  25%|██▌       | 79008/315947 [00:01<04:25, 892.52 examples/s]
Saving the dataset (49/192 shards): 
3:  26%|██▌       | 80654/315947 [00:01<04:23, 892.52 examples/s]
Saving the dataset (50/192 shards):  26%|██▌       | 82300/315947 [00:01<04:21, 892.52 examples/s]
Saving the dataset (51/192 shards):  27%|██▋       | 83946/315947 [00:01<04:19, 892.52 examples/s]
Saving the dataset (52/192 shards):  27%|██▋       | 85592/315947 [00:01<04:18, 892.52 examples/s]
Saving the dataset (53/192 shards):  28%|██▊       | 87238/315947 [00:01<04:16, 892.52 examples/s]
Saving the dataset (54/192 shards):  28%|██▊       | 88884/315947 [00:01<04:14, 892.52 examples/s]
Saving the dataset (55/192 shards):  29%|██▉       | 92176/315947 [00:01<04:10, 892.52 examples/s]
Saving the dataset (56/192 shards):  29%|██▉       | 92176/315947 [00:01<04:10, 892.52 examples/s]
Saving the dataset (57/192 shards):  30%|██▉       | 93822/315947 [00:01<04:08, 892.52 examples/s]
Saving the dataset (58/192 shards):  30%|███       | 95468/315947 [00:01<04:07, 892.52 examples/s]
Saving the
3:  dataset (59/192 shards):  32%|███▏      | 100406/315947 [00:01<04:01, 892.52 examples/s]
Saving the dataset (60/192 shards):  32%|███▏      | 100406/315947 [00:01<04:01, 892.52 examples/s]
Saving the dataset (61/192 shards):  32%|███▏      | 100406/315947 [00:01<04:01, 892.52 examples/s]
Saving the dataset (62/192 shards):  32%|███▏      | 102052/315947 [00:01<03:59, 892.52 examples/s]
Saving the dataset (63/192 shards):  33%|███▎      | 103698/315947 [00:01<03:57, 892.52 examples/s]
Saving the dataset (64/192 shards):  34%|███▍      | 106990/315947 [00:01<03:54, 892.52 examples/s]
Saving the dataset (65/192 shards):  34%|███▍      | 106990/315947 [00:01<03:54, 892.52 examples/s]
Saving the dataset (66/192 shards):  34%|███▍      | 108636/315947 [00:01<03:52, 892.52 examples/s]
Saving the dataset (67/192 shards):  36%|███▌      | 113574/315947 [00:01<03:46, 892.52 examples/s]
Saving the dataset (68/192 shards):  36%|███▌      | 1
3: 13574/315947 [00:01<03:46, 892.52 examples/s]
Saving the dataset (69/192 shards):  36%|███▌      | 113574/315947 [00:01<03:46, 892.52 examples/s]
Saving the dataset (70/192 shards):  36%|███▋      | 115220/315947 [00:01<03:44, 892.52 examples/s]
Saving the dataset (71/192 shards):  38%|███▊      | 118512/315947 [00:01<03:41, 892.52 examples/s]
Saving the dataset (72/192 shards):  38%|███▊      | 118512/315947 [00:01<03:41, 892.52 examples/s]
Saving the dataset (73/192 shards):  39%|███▊      | 121804/315947 [00:01<03:37, 892.52 examples/s]
Saving the dataset (74/192 shards):  39%|███▊      | 121804/315947 [00:01<03:37, 892.52 examples/s]
Saving the dataset (75/192 shards):  39%|███▉      | 123450/315947 [00:01<03:35, 892.52 examples/s]
Saving the dataset (76/192 shards):  40%|███▉      | 125096/315947 [00:01<03:33, 892.52 examples/s]
Saving the dataset (77/192 shards):  40%|████      | 126742/315947 [00:01<03:31, 892.52 examples/s]
Saving
3:  the dataset (78/192 shards):  41%|████      | 130034/315947 [00:01<03:28, 892.52 examples/s]
Saving the dataset (79/192 shards):  41%|████      | 130034/315947 [00:01<03:28, 892.52 examples/s]
Saving the dataset (80/192 shards):  42%|████▏     | 131680/315947 [00:01<03:26, 892.52 examples/s]
Saving the dataset (81/192 shards):  42%|████▏     | 133326/315947 [00:01<03:24, 892.52 examples/s]
Saving the dataset (82/192 shards):  43%|████▎     | 134972/315947 [00:01<03:22, 892.52 examples/s]
Saving the dataset (83/192 shards):  43%|████▎     | 136618/315947 [00:01<03:20, 892.52 examples/s]
Saving the dataset (84/192 shards):  44%|████▍     | 138264/315947 [00:01<03:19, 892.52 examples/s]
Saving the dataset (85/192 shards):  44%|████▍     | 139910/315947 [00:01<03:17, 892.52 examples/s]
Saving the dataset (86/192 shards):  45%|████▍     | 141556/315947 [00:01<03:15, 892.52 examples/s]
Saving the dataset (87/192 shards):  46%|█
3: ███▌     | 144848/315947 [00:01<03:11, 892.52 examples/s]
Saving the dataset (88/192 shards):  46%|████▌     | 144848/315947 [00:01<03:11, 892.52 examples/s]
Saving the dataset (89/192 shards):  46%|████▋     | 146494/315947 [00:01<03:09, 892.52 examples/s]
Saving the dataset (90/192 shards):  47%|████▋     | 148140/315947 [00:01<03:08, 892.52 examples/s]
Saving the dataset (91/192 shards):  47%|████▋     | 149786/315947 [00:01<03:06, 892.52 examples/s]
Saving the dataset (92/192 shards):  48%|████▊     | 151432/315947 [00:01<03:04, 892.52 examples/s]
Saving the dataset (93/192 shards):  48%|████▊     | 153078/315947 [00:01<03:02, 892.52 examples/s]
Saving the dataset (94/192 shards):  49%|████▉     | 154724/315947 [00:01<03:00, 892.52 examples/s]
Saving the dataset (95/192 shards):  50%|█████     | 158016/315947 [00:01<02:56, 892.52 examples/s]
Saving the dataset (96/192 shards):  50%|█████     | 158016/315947 [
3: 00:01<02:56, 892.52 examples/s]
Saving the dataset (97/192 shards):  51%|█████     | 159662/315947 [00:01<02:55, 892.52 examples/s]
Saving the dataset (98/192 shards):  51%|█████     | 161308/315947 [00:01<02:53, 892.52 examples/s]
Saving the dataset (99/192 shards):  52%|█████▏    | 162954/315947 [00:01<02:51, 892.52 examples/s]
Saving the dataset (100/192 shards):  52%|█████▏    | 164600/315947 [00:01<02:49, 892.52 examples/s]
Saving the dataset (101/192 shards):  53%|█████▎    | 166246/315947 [00:01<02:47, 892.52 examples/s]
Saving the dataset (102/192 shards):  53%|█████▎    | 167892/315947 [00:01<02:45, 892.52 examples/s]
Saving the dataset (103/192 shards):  54%|█████▍    | 171184/315947 [00:01<02:42, 892.52 examples/s]
Saving the dataset (104/192 shards):  54%|█████▍    | 171184/315947 [00:01<02:42, 892.52 examples/s]
Saving the dataset (105/192 shards):  55%|█████▍    | 172830/315947 [00:01<02:40, 8
3: 92.52 examples/s]
Saving the dataset (106/192 shards):  55%|█████▌    | 174476/315947 [00:01<02:38, 892.52 examples/s]
Saving the dataset (107/192 shards):  56%|█████▌    | 176122/315947 [00:01<02:36, 892.52 examples/s]
Saving the dataset (108/192 shards):  56%|█████▋    | 177767/315947 [00:01<02:34, 892.52 examples/s]
Saving the dataset (109/192 shards):  57%|█████▋    | 179412/315947 [00:01<02:32, 892.52 examples/s]
Saving the dataset (110/192 shards):  58%|█████▊    | 182702/315947 [00:01<02:29, 892.52 examples/s]
Saving the dataset (111/192 shards):  58%|█████▊    | 182702/315947 [00:01<02:29, 892.52 examples/s]
Saving the dataset (112/192 shards):  58%|█████▊    | 184347/315947 [00:01<02:27, 892.52 examples/s]
Saving the dataset (113/192 shards):  59%|█████▉    | 187637/315947 [00:01<02:23, 892.52 examples/s]
Saving the dataset (114/192 shards):  59%|█████▉    | 187637/315947 [00:01<02:23, 892.52 e
3: xamples/s]
Saving the dataset (115/192 shards):  60%|█████▉    | 189282/315947 [00:01<02:21, 892.52 examples/s]
Saving the dataset (116/192 shards):  60%|██████    | 190927/315947 [00:01<02:20, 892.52 examples/s]
Saving the dataset (117/192 shards):  61%|██████    | 192572/315947 [00:01<02:18, 892.52 examples/s]
Saving the dataset (118/192 shards):  61%|██████▏   | 194217/315947 [00:01<02:16, 892.52 examples/s]
Saving the dataset (119/192 shards):  62%|██████▏   | 195862/315947 [00:01<02:14, 892.52 examples/s]
Saving the dataset (120/192 shards):  63%|██████▎   | 197507/315947 [00:01<02:12, 892.52 examples/s]
Saving the dataset (121/192 shards):  64%|██████▎   | 200797/315947 [00:01<02:09, 892.52 examples/s]
Saving the dataset (122/192 shards):  64%|██████▎   | 200797/315947 [00:01<02:09, 892.52 examples/s]
Saving the dataset (123/192 shards):  64%|██████▍   | 202442/315947 [00:01<02:07, 892
3: .52 examples/s]
Saving the dataset (124/192 shards):  65%|██████▍   | 204087/315947 [00:01<02:05, 892.52 examples/s]
Saving the dataset (125/192 shards):  65%|██████▌   | 205732/315947 [00:01<02:03, 892.52 examples/s]
Saving the dataset (126/192 shards):  66%|██████▌   | 207377/315947 [00:01<02:01, 892.52 examples/s]
Saving the dataset (127/192 shards):  66%|██████▌   | 209022/315947 [00:01<01:59, 892.52 examples/s]
Saving the dataset (128/192 shards):  67%|██████▋   | 210667/315947 [00:01<01:57, 892.52 examples/s]
Saving the dataset (129/192 shards):  67%|██████▋   | 212312/315947 [00:01<01:56, 892.52 examples/s]
Saving the dataset (130/192 shards):  68%|██████▊   | 213957/315947 [00:01<01:54, 892.52 examples/s]
Saving the dataset (131/192 shards):  68%|██████▊   | 215602/315947 [00:01<01:52, 892.52 examples/s]
Saving the dataset (132/192 shards):  69%|██████▉   | 217247/315947 [00:01
3: <01:50, 892.52 examples/s]
Saving the dataset (133/192 shards):  69%|██████▉   | 218892/315947 [00:01<01:48, 892.52 examples/s]
Saving the dataset (134/192 shards):  70%|██████▉   | 220537/315947 [00:01<01:46, 892.52 examples/s]
Saving the dataset (135/192 shards):  70%|███████   | 222182/315947 [00:01<01:45, 892.52 examples/s]
Saving the dataset (136/192 shards):  71%|███████   | 223827/315947 [00:01<01:43, 892.52 examples/s]
Saving the dataset (137/192 shards):  71%|███████▏  | 225472/315947 [00:01<01:41, 892.52 examples/s]
Saving the dataset (138/192 shards):  72%|███████▏  | 227117/315947 [00:01<01:39, 892.52 examples/s]
Saving the dataset (139/192 shards):  72%|███████▏  | 228762/315947 [00:01<01:37, 892.52 examples/s]
Saving the dataset (140/192 shards):  73%|███████▎  | 230407/315947 [00:01<01:35, 892.52 examples/s]
Saving the dataset (141/192 shards):  73%|███████▎  |
3:  232052/315947 [00:01<01:33, 892.52 examples/s]
Saving the dataset (142/192 shards):  74%|███████▍  | 233697/315947 [00:01<01:32, 892.52 examples/s]
Saving the dataset (143/192 shards):  74%|███████▍  | 235342/315947 [00:01<01:30, 892.52 examples/s]
Saving the dataset (144/192 shards):  75%|███████▌  | 236987/315947 [00:01<01:28, 892.52 examples/s]
Saving the dataset (145/192 shards):  76%|███████▌  | 238632/315947 [00:01<01:26, 892.52 examples/s]
Saving the dataset (146/192 shards):  76%|███████▌  | 240277/315947 [00:01<01:24, 892.52 examples/s]
Saving the dataset (147/192 shards):  77%|███████▋  | 241922/315947 [00:01<01:22, 892.52 examples/s]
Saving the dataset (148/192 shards):  77%|███████▋  | 243567/315947 [00:01<01:21, 892.52 examples/s]
Saving the dataset (149/192 shards):  78%|███████▊  | 245212/315947 [00:01<01:19, 892.52 examples/s]
Saving the dataset (150/192 shards):  78
3: %|███████▊  | 246857/315947 [00:01<01:17, 892.52 examples/s]
Saving the dataset (151/192 shards):  79%|███████▊  | 248502/315947 [00:01<01:15, 892.52 examples/s]
Saving the dataset (152/192 shards):  80%|███████▉  | 251792/315947 [00:01<01:11, 892.52 examples/s]
Saving the dataset (153/192 shards):  80%|███████▉  | 251792/315947 [00:01<01:11, 892.52 examples/s]
Saving the dataset (154/192 shards):  80%|████████  | 253437/315947 [00:01<01:10, 892.52 examples/s]
Saving the dataset (155/192 shards):  81%|████████  | 255082/315947 [00:01<01:08, 892.52 examples/s]
Saving the dataset (156/192 shards):  81%|████████▏ | 256727/315947 [00:01<01:06, 892.52 examples/s]
Saving the dataset (157/192 shards):  82%|████████▏ | 258372/315947 [00:01<01:04, 892.52 examples/s]
Saving the dataset (158/192 shards):  82%|████████▏ | 260017/315947 [00:01<01:02, 892.52 examples/s]
Savin
3: g the dataset (159/192 shards):  83%|████████▎ | 261662/315947 [00:01<01:00, 892.52 examples/s]
Saving the dataset (160/192 shards):  83%|████████▎ | 263307/315947 [00:01<00:58, 892.52 examples/s]
Saving the dataset (161/192 shards):  84%|████████▍ | 264952/315947 [00:01<00:57, 892.52 examples/s]
Saving the dataset (162/192 shards):  84%|████████▍ | 266597/315947 [00:01<00:55, 892.52 examples/s]
Saving the dataset (163/192 shards):  85%|████████▍ | 268242/315947 [00:01<00:53, 892.52 examples/s]
Saving the dataset (164/192 shards):  85%|████████▌ | 269887/315947 [00:01<00:51, 892.52 examples/s]
Saving the dataset (165/192 shards):  86%|████████▌ | 271532/315947 [00:01<00:49, 892.52 examples/s]
Saving the dataset (166/192 shards):  86%|████████▋ | 273177/315947 [00:01<00:47, 892.52 examples/s]
Saving the dataset (167/192 shards):  87%|████████▋ | 27482
3: 2/315947 [00:01<00:46, 892.52 examples/s]
Saving the dataset (168/192 shards):  88%|████████▊ | 276467/315947 [00:01<00:44, 892.52 examples/s]
Saving the dataset (169/192 shards):  88%|████████▊ | 278112/315947 [00:01<00:42, 892.52 examples/s]
Saving the dataset (170/192 shards):  89%|████████▊ | 279757/315947 [00:01<00:40, 892.52 examples/s]
Saving the dataset (171/192 shards):  89%|████████▉ | 281402/315947 [00:01<00:38, 892.52 examples/s]
Saving the dataset (172/192 shards):  90%|████████▉ | 283047/315947 [00:01<00:36, 892.52 examples/s]
Saving the dataset (173/192 shards):  90%|█████████ | 284692/315947 [00:01<00:35, 892.52 examples/s]
Saving the dataset (174/192 shards):  91%|█████████ | 286337/315947 [00:01<00:33, 892.52 examples/s]
Saving the dataset (175/192 shards):  91%|█████████ | 287982/315947 [00:01<00:31, 892.52 examples/s]
Saving the dataset (176/192 sh
3: ards):  92%|█████████▏| 289627/315947 [00:01<00:29, 892.52 examples/s]
Saving the dataset (177/192 shards):  92%|█████████▏| 291272/315947 [00:01<00:27, 892.52 examples/s]
Saving the dataset (178/192 shards):  93%|█████████▎| 292917/315947 [00:01<00:25, 892.52 examples/s]
Saving the dataset (179/192 shards):  93%|█████████▎| 294562/315947 [00:01<00:23, 892.52 examples/s]
Saving the dataset (180/192 shards):  94%|█████████▍| 296207/315947 [00:01<00:22, 892.52 examples/s]
Saving the dataset (181/192 shards):  94%|█████████▍| 297852/315947 [00:01<00:20, 892.52 examples/s]
Saving the dataset (182/192 shards):  95%|█████████▍| 299497/315947 [00:01<00:18, 892.52 examples/s]
Saving the dataset (183/192 shards):  95%|█████████▌| 301142/315947 [00:01<00:16, 892.52 examples/s]
Saving the dataset (184/192 shards):  96%|█████████▌| 302787/31594
3: 7 [00:01<00:14, 892.52 examples/s]
Saving the dataset (185/192 shards):  96%|█████████▋| 304432/315947 [00:01<00:12, 892.52 examples/s]
Saving the dataset (186/192 shards):  97%|█████████▋| 307722/315947 [00:01<00:09, 892.52 examples/s]
Saving the dataset (187/192 shards):  97%|█████████▋| 307722/315947 [00:01<00:09, 892.52 examples/s]
Saving the dataset (188/192 shards):  98%|█████████▊| 309367/315947 [00:01<00:07, 892.52 examples/s]
Saving the dataset (189/192 shards):  98%|█████████▊| 311012/315947 [00:01<00:05, 892.52 examples/s]
Saving the dataset (190/192 shards):  99%|█████████▉| 312657/315947 [00:01<00:03, 892.52 examples/s]
Saving the dataset (191/192 shards):  99%|█████████▉| 314302/315947 [00:01<00:01, 892.52 examples/s]
Saving the dataset (192/192 shards): 100%|██████████| 315947/315947 [00:01<00:00, 892.52 examples/s]
Saving the dataset (1
3: 92/192 shards): 100%|██████████| 315947/315947 [00:01<00:00, 158524.02 examples/s]
0: [2025-09-02 19:00:29,431] [INFO] [axolotl.utils.data.shared.load_preprocessed_dataset:472] [PID:2134977] [RANK:0] Loading prepared dataset from disk at /lustre/fsn1/projects/rech/dgo/udv55np/dataset_math/Qwen3-235B-A22B/0/8bdfa514f4bdb23bcb10d93ba101ff0f...
0: [2025-09-02 19:01:04,779] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:436] [PID:2134977] [RANK:0] gather_len_batches: [25939, 25939, 25939, 25939, 25940, 25939, 25939, 25940, 25939, 25937, 25939, 25940, 25940, 25939, 25939, 25939]
0: [2025-09-02 19:01:04,844] [INFO] [axolotl.utils.trainer.calc_sample_packing_eff_est:495] [PID:2134977] [RANK:0] sample_packing_eff_est across ranks: [0.9965166449546814, 0.9965550303459167, 0.9965550303459167, 0.996631920337677, 0.9965934753417969, 0.9964781999588013, 0.9965166449546814, 0.9965934753417969, 0.9965934753417969, 0.9965934753417969, 0.9965550303459167, 0.9965550303459167, 0.9965550303459167, 0.9965550303459167, 0.9965550303459167, 0.9965550303459167]
0: [2025-09-02 19:01:04,850] [INFO] [axolotl.utils.data.sft._prepare_standard_dataset:127] [PID:2134977] [RANK:0] Maximum number of steps set at 1621
0: [2025-09-02 19:01:05,194] [INFO] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_evaluation_loop:110] [PID:2134977] [RANK:0] Patched Trainer.evaluation_loop with nanmean loss calculation
0: [2025-09-02 19:01:05,195] [INFO] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_maybe_log_save_evaluate:164] [PID:2134977] [RANK:0] Patched Trainer._maybe_log_save_evaluate with nanmean loss calculation
0: 
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:36,  7.20s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:36,  7.21s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:36,  7.20s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:36,  7.31s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.62s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.62s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.62s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.66s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:28<00:29,  9.77s/it]
Loading checkpoint shards:  50%|█████ 
2: 
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:35,  7.20s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:36,  7.20s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:35,  7.20s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:36,  7.20s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.62s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.62s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.62s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.62s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:28<00:29,  9.77s/it]
Loading checkpoint shards:  50%|█████ 
3: 
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:36,  7.23s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:36,  7.23s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:36,  7.23s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:36,  7.23s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.63s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.63s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.63s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.63s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:28<00:29,  9.78s/it]
Loading checkpoint shards:  50%|█████ 
1: 
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/6 [00:00<?, ?it/s]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:35,  7.19s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:35,  7.19s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:35,  7.19s/it]
Loading checkpoint shards:  17%|█▋        | 1/6 [00:07<00:35,  7.19s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.61s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.62s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.61s/it]
Loading checkpoint shards:  33%|███▎      | 2/6 [00:18<00:38,  9.62s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:28<00:29,  9.77s/it]
Loading checkpoint shards:  50%|█████ 
0:     | 3/6 [00:28<00:29,  9.77s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:28<00:29,  9.77s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:28<00:29,  9.79s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.97s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.97s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.97s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.99s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.53s/it]
Loading checkpoint
0:  shards: 100%|██████████| 6/6 [00:57<00:00,  9.53s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.66s/it]
0: 
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.66s/it]
2:     | 3/6 [00:28<00:29,  9.77s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:28<00:29,  9.77s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:28<00:29,  9.77s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.97s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.97s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.97s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.97s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.53s/it]
Loading checkpoint
2:  shards: 100%|██████████| 6/6 [00:57<00:00,  9.53s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.66s/it]
1:     | 3/6 [00:28<00:29,  9.77s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:28<00:29,  9.77s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:28<00:29,  9.77s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.97s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.97s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.97s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.97s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.53s/it]
Loading checkpoint
2: 
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.66s/it]
1:  shards: 100%|██████████| 6/6 [00:57<00:00,  9.53s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.65s/it]
1: 
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.65s/it]
2: 
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.53s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.66s/it]
3:     | 3/6 [00:28<00:29,  9.78s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:28<00:29,  9.78s/it]
Loading checkpoint shards:  50%|█████     | 3/6 [00:28<00:29,  9.78s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.98s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.98s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.98s/it]
Loading checkpoint shards:  67%|██████▋   | 4/6 [00:38<00:19,  9.98s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards:  83%|████████▎ | 5/6 [00:50<00:10, 10.62s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.54s/it]
Loading checkpoint
0: 
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.53s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.66s/it]
1: 
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.53s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.65s/it]
3:  shards: 100%|██████████| 6/6 [00:57<00:00,  9.54s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.66s/it]
3: 
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.66s/it]
3: 
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.54s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.66s/it]
1: 
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.53s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.65s/it]
3: 
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.54s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.66s/it]
2: 
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.53s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:57<00:00,  9.66s/it]
0: 
Loading checkpoint shards: 100%|██████████| 6/6 [00:59<00:00, 10.18s/it]
Loading checkpoint shards: 100%|██████████| 6/6 [00:59<00:00,  9.99s/it]
0: [2025-09-02 19:02:05,501] [INFO] [axolotl.loaders.model._configure_embedding_dtypes:345] [PID:2134977] [RANK:0] Converting modules to torch.bfloat16
0: [2025-09-02 19:02:09,288] [INFO] [axolotl.train.save_initial_configs:416] [PID:2134977] [RANK:0] Pre-saving tokenizer to /lustre/fswork/projects/rech/dgo/udv55np/math/Qwen3-235B-A22B/Qwen2.5-14B_ift/0...
0: [2025-09-02 19:02:09,487] [INFO] [axolotl.train.save_initial_configs:419] [PID:2134977] [RANK:0] Pre-saving model config to /lustre/fswork/projects/rech/dgo/udv55np/math/Qwen3-235B-A22B/Qwen2.5-14B_ift/0...
0: [2025-09-02 19:02:09,502] [INFO] [axolotl.train.execute_training:203] [PID:2134977] [RANK:0] Starting trainer...
0: [2025-09-02 19:04:11,579] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:436] [PID:2134977] [RANK:0] gather_len_batches: [25939, 25939, 25939, 25939, 25939, 25939, 25939, 25939, 25939, 25939, 25939, 25939, 25939, 25939, 25939, 25939]
0: Parameter Offload - Persistent parameters statistics: param_count = 241, numel = 840704
0: {'loss': 0.2404, 'grad_norm': 0.364680268021655, 'learning_rate': 1.54e-07, 'memory/max_mem_active(gib)': 52.3, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.01}
0: 
  0%|          | 0/1621 [00:00<?, ?it/s]
  0%|          | 1/1621 [03:16<88:13:23, 196.05s/it]
  0%|          | 2/1621 [03:20<37:32:36, 83.48s/it] 
  0%|          | 3/1621 [03:25<21:22:10, 47.55s/it]
  0%|          | 4/1621 [03:30<13:44:38, 30.60s/it]
  0%|          | 5/1621 [03:35<9:35:07, 21.35s/it] 
  0%|          | 6/1621 [03:39<7:02:38, 15.70s/it]
  0%|          | 7/1621 [03:45<5:33:21, 12.39s/it]
  0%|          | 8/1621 [03:50<4:27:23,  9.95s/it]
  1%|          | 9/1621 [03:54<3:42:49,  8.29s/it]
  1%|          | 10/1621 [04:00<3:17:19,  7.35s/it]
                                                   

  1%|          | 10/1621 [04:00<3:17:19,  7.35s/it]
  1%|          | 11/1621 [04:05<2:58:11,  6.64s/it]
  1%|          | 12/1621 [04:09<2:42:25,  6.06s/it]
  1%|          | 13/1621 [04:14<2:32:10,  5.68s/it]
  1%|          | 14/1621 [04:19<2:24:06,  5.38s/it]
  1%|          | 15/1621 [04:24<2:23:35,  5.36s/it]
  1%|          | 16/1621 [04:29<2:18:24,  5.17s/it]
  1%|          | 17/1621 [04:34<2:16:57,  5.12s
0: {'loss': 0.229, 'grad_norm': 0.29208771571602604, 'learning_rate': 2.14e-07, 'memory/max_mem_active(gib)': 52.3, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.01}
0: {'loss': 0.2364, 'grad_norm': 0.3147847192135592, 'learning_rate': 2.74e-07, 'memory/max_mem_active(gib)': 53.83, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.02}
0: /it]
  1%|          | 18/1621 [04:39<2:13:20,  4.99s/it]
  1%|          | 19/1621 [04:43<2:10:48,  4.90s/it]
  1%|          | 20/1621 [04:48<2:09:49,  4.87s/it]
                                                   

  1%|          | 20/1621 [04:48<2:09:49,  4.87s/it]
  1%|▏         | 21/1621 [04:54<2:17:13,  5.15s/it]
  1%|▏         | 22/1621 [04:59<2:14:38,  5.05s/it]
  1%|▏         | 23/1621 [05:03<2:12:25,  4.97s/it]
  1%|▏         | 24/1621 [05:08<2:10:25,  4.90s/it]
  2%|▏         | 25/1621 [05:13<2:08:46,  4.84s/it]
  2%|▏         | 26/1621 [05:18<2:07:33,  4.80s/it]
  2%|▏         | 27/1621 [05:23<2:09:47,  4.89s/it]
  2%|▏         | 28/1621 [05:27<2:09:07,  4.86s/it]
  2%|▏         | 29/1621 [05:32<2:07:33,  4.81s/it]
  2%|▏         | 30/1621 [05:37<2:06:00,  4.75s/it]
                                                   

  2%|▏         | 30/1621 [05:37<2:06:00,  4.75s/it]
  2%|▏         | 31/1621 [05:42<2:06:18,  4.77s/it]
  2%|▏         | 32/1621 [05:47<2:07:55,  4.83s/it]
  2
0: {'loss': 0.2316, 'grad_norm': 0.2898188486284449, 'learning_rate': 3.34e-07, 'memory/max_mem_active(gib)': 53.83, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.02}
0: %|▏         | 33/1621 [05:51<2:06:38,  4.78s/it]
  2%|▏         | 34/1621 [05:56<2:08:50,  4.87s/it]
  2%|▏         | 35/1621 [06:01<2:08:43,  4.87s/it]
  2%|▏         | 36/1621 [06:06<2:07:47,  4.84s/it]
  2%|▏         | 37/1621 [06:11<2:07:14,  4.82s/it]
  2%|▏         | 38/1621 [06:16<2:13:26,  5.06s/it]
  2%|▏         | 39/1621 [06:21<2:10:44,  4.96s/it]
  2%|▏         | 40/1621 [06:26<2:09:23,  4.91s/it]
                                                   

  2%|▏         | 40/1621 [06:26<2:09:23,  4.91s/it]
  3%|▎         | 41/1621 [06:31<2:09:26,  4.92s/it]
  3%|▎         | 42/1621 [06:36<2:09:02,  4.90s/it]
  3%|▎         | 43/1621 [06:41<2:08:48,  4.90s/it]
  3%|▎         | 44/1621 [06:47<2:17:38,  5.24s/it]
  3%|▎         | 45/1621 [06:51<2:13:22,  5.08s/it]
  3%|▎         | 46/1621 [06:56<2:13:42,  5.09s/it]
  3%|▎         | 47/1621 [07:03<2:22:28,  5.43s/it]
  3%|▎         | 48/1621 [07:08<2:18:54,  5.30s/it]
  3%|▎         | 49/1621 [07:12<2:15:19,  5.17s/it]
  
0: {'loss': 0.2356, 'grad_norm': 0.30174914984263745, 'learning_rate': 3.94e-07, 'memory/max_mem_active(gib)': 53.83, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.03}
0: {'loss': 0.2328, 'grad_norm': 0.39057298409934055, 'learning_rate': 4.5399999999999996e-07, 'memory/max_mem_active(gib)': 53.83, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.04}
0: 3%|▎         | 50/1621 [07:17<2:11:06,  5.01s/it]
                                                   

  3%|▎         | 50/1621 [07:17<2:11:06,  5.01s/it]
  3%|▎         | 51/1621 [07:22<2:09:59,  4.97s/it]
  3%|▎         | 52/1621 [07:27<2:11:56,  5.05s/it]
  3%|▎         | 53/1621 [07:32<2:10:03,  4.98s/it]
  3%|▎         | 54/1621 [07:37<2:07:16,  4.87s/it]
  3%|▎         | 55/1621 [07:41<2:05:36,  4.81s/it]
  3%|▎         | 56/1621 [07:47<2:09:47,  4.98s/it]
  4%|▎         | 57/1621 [07:52<2:14:39,  5.17s/it]
  4%|▎         | 58/1621 [07:57<2:11:09,  5.04s/it]
  4%|▎         | 59/1621 [08:02<2:09:51,  4.99s/it]
  4%|▎         | 60/1621 [08:07<2:08:00,  4.92s/it]
                                                   

  4%|▎         | 60/1621 [08:07<2:08:00,  4.92s/it]
  4%|▍         | 61/1621 [08:12<2:08:12,  4.93s/it]
  4%|▍         | 62/1621 [08:16<2:06:51,  4.88s/it]
  4%|▍         | 63/1621 [08:21<2:05:44,  4.84s/it]
  4%|▍         | 64/1621 [08:26<2:05:05,  4.82s/it]
  
0: {'loss': 0.2363, 'grad_norm': 0.4675017545126673, 'learning_rate': 5.14e-07, 'memory/max_mem_active(gib)': 53.83, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.04}
0: {'loss': 0.2345, 'grad_norm': 0.2788174122742784, 'learning_rate': 5.739999999999999e-07, 'memory/max_mem_active(gib)': 53.83, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.05}
0: 4%|▍         | 65/1621 [08:31<2:10:08,  5.02s/it]
  4%|▍         | 66/1621 [08:36<2:08:23,  4.95s/it]
  4%|▍         | 67/1621 [08:41<2:07:03,  4.91s/it]
  4%|▍         | 68/1621 [08:46<2:05:37,  4.85s/it]
  4%|▍         | 69/1621 [08:50<2:04:01,  4.79s/it]
  4%|▍         | 70/1621 [08:55<2:02:54,  4.75s/it]
                                                   

  4%|▍         | 70/1621 [08:55<2:02:54,  4.75s/it]
  4%|▍         | 71/1621 [09:00<2:03:44,  4.79s/it]
  4%|▍         | 72/1621 [09:05<2:03:06,  4.77s/it]
  5%|▍         | 73/1621 [09:09<2:03:25,  4.78s/it]
  5%|▍         | 74/1621 [09:15<2:07:31,  4.95s/it]
  5%|▍         | 75/1621 [09:19<2:05:23,  4.87s/it]
  5%|▍         | 76/1621 [09:24<2:03:29,  4.80s/it]
  5%|▍         | 77/1621 [09:29<2:03:04,  4.78s/it]
  5%|▍         | 78/1621 [09:34<2:03:39,  4.81s/it]
  5%|▍         | 79/1621 [09:38<2:02:28,  4.77s/it]
  5%|▍         | 80/1621 [09:43<2:01:28,  4.73s/it]
                                                   

  
0: {'loss': 0.2368, 'grad_norm': 0.2746391832738924, 'learning_rate': 6.34e-07, 'memory/max_mem_active(gib)': 53.83, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.06}
0: 5%|▍         | 80/1621 [09:43<2:01:28,  4.73s/it]
  5%|▍         | 81/1621 [09:48<2:01:03,  4.72s/it]
  5%|▌         | 82/1621 [09:53<2:06:03,  4.91s/it]
  5%|▌         | 83/1621 [09:58<2:04:16,  4.85s/it]
  5%|▌         | 84/1621 [10:02<2:03:08,  4.81s/it]
  5%|▌         | 85/1621 [10:07<2:03:24,  4.82s/it]
  5%|▌         | 86/1621 [10:12<2:02:23,  4.78s/it]
  5%|▌         | 87/1621 [10:17<2:01:48,  4.76s/it]
  5%|▌         | 88/1621 [10:21<2:00:55,  4.73s/it]
  5%|▌         | 89/1621 [10:26<2:00:53,  4.73s/it]
  6%|▌         | 90/1621 [10:31<2:00:48,  4.73s/it]
                                                   

  6%|▌         | 90/1621 [10:31<2:00:48,  4.73s/it]
  6%|▌         | 91/1621 [10:36<2:00:21,  4.72s/it]
  6%|▌         | 92/1621 [10:41<2:09:11,  5.07s/it]
  6%|▌         | 93/1621 [10:46<2:07:23,  5.00s/it]
  6%|▌         | 94/1621 [10:52<2:12:01,  5.19s/it]
  6%|▌         | 95/1621 [10:57<2:09:02,  5.07s/it]
  6%|▌         | 96/1621 [11:01<2:05:57,  4.96s/it]
 
0: {'loss': 0.2293, 'grad_norm': 0.29838762735346686, 'learning_rate': 6.94e-07, 'memory/max_mem_active(gib)': 53.83, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.06}
0: {'loss': 0.231, 'grad_norm': 0.275344655730642, 'learning_rate': 7.54e-07, 'memory/max_mem_active(gib)': 53.83, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.07}
0:  6%|▌         | 97/1621 [11:07<2:09:45,  5.11s/it]
  6%|▌         | 98/1621 [11:12<2:08:02,  5.04s/it]
  6%|▌         | 99/1621 [11:17<2:09:26,  5.10s/it]
  6%|▌         | 100/1621 [11:22<2:07:20,  5.02s/it]
                                                    

  6%|▌         | 100/1621 [11:22<2:07:20,  5.02s/it]
  6%|▌         | 101/1621 [11:27<2:06:04,  4.98s/it]
  6%|▋         | 102/1621 [11:31<2:04:16,  4.91s/it]
  6%|▋         | 103/1621 [11:37<2:05:48,  4.97s/it]
  6%|▋         | 104/1621 [11:41<2:04:00,  4.90s/it]
  6%|▋         | 105/1621 [11:46<2:04:38,  4.93s/it]
  7%|▋         | 106/1621 [11:51<2:03:31,  4.89s/it]
  7%|▋         | 107/1621 [11:56<2:04:36,  4.94s/it]
  7%|▋         | 108/1621 [12:02<2:09:36,  5.14s/it]
  7%|▋         | 109/1621 [12:06<2:06:22,  5.01s/it]
  7%|▋         | 110/1621 [12:12<2:07:28,  5.06s/it]
                                                    

  7%|▋         | 110/1621 [12:12<2:07:28,  5.06s/it]
  7%|▋         | 111/1621 [12:17<2:06:
0: {'loss': 0.2313, 'grad_norm': 0.2697336383234993, 'learning_rate': 8.14e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.07}
0: 48,  5.04s/it]
  7%|▋         | 112/1621 [12:22<2:09:10,  5.14s/it]
  7%|▋         | 113/1621 [12:27<2:06:01,  5.01s/it]
  7%|▋         | 114/1621 [12:31<2:04:00,  4.94s/it]
  7%|▋         | 115/1621 [12:36<2:01:55,  4.86s/it]
  7%|▋         | 116/1621 [12:41<2:02:07,  4.87s/it]
  7%|▋         | 117/1621 [12:46<2:01:10,  4.83s/it]
  7%|▋         | 118/1621 [12:51<2:00:22,  4.81s/it]
  7%|▋         | 119/1621 [12:55<2:00:09,  4.80s/it]
  7%|▋         | 120/1621 [13:00<2:01:29,  4.86s/it]
                                                    

  7%|▋         | 120/1621 [13:00<2:01:29,  4.86s/it]
  7%|▋         | 121/1621 [13:05<2:00:49,  4.83s/it]
  8%|▊         | 122/1621 [13:10<1:59:47,  4.79s/it]
  8%|▊         | 123/1621 [13:15<1:59:43,  4.80s/it]
  8%|▊         | 124/1621 [13:19<1:59:46,  4.80s/it]
  8%|▊         | 125/1621 [13:25<2:07:20,  5.11s/it]
  8%|▊         | 126/1621 [13:30<2:06:59,  5.10s/it]
  8%|▊         | 127/1621 [13:35<2:04:05,  4.98s/it]
  8%|▊         | 1
0: {'loss': 0.2332, 'grad_norm': 0.2602617441366092, 'learning_rate': 8.739999999999999e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.08}
0: {'loss': 0.2302, 'grad_norm': 0.29568418904442645, 'learning_rate': 9.339999999999999e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.09}
0: 28/1621 [13:40<2:02:36,  4.93s/it]
  8%|▊         | 129/1621 [13:44<2:00:34,  4.85s/it]
  8%|▊         | 130/1621 [13:50<2:02:56,  4.95s/it]
                                                    

  8%|▊         | 130/1621 [13:50<2:02:56,  4.95s/it]
  8%|▊         | 131/1621 [13:55<2:07:42,  5.14s/it]
  8%|▊         | 132/1621 [14:00<2:06:36,  5.10s/it]
  8%|▊         | 133/1621 [14:06<2:07:58,  5.16s/it]
  8%|▊         | 134/1621 [14:10<2:04:47,  5.04s/it]
  8%|▊         | 135/1621 [14:16<2:09:02,  5.21s/it]
  8%|▊         | 136/1621 [14:21<2:04:56,  5.05s/it]
  8%|▊         | 137/1621 [14:25<2:03:24,  4.99s/it]
  9%|▊         | 138/1621 [14:31<2:09:53,  5.26s/it]
  9%|▊         | 139/1621 [14:37<2:09:39,  5.25s/it]
  9%|▊         | 140/1621 [14:41<2:05:50,  5.10s/it]
                                                    

  9%|▊         | 140/1621 [14:41<2:05:50,  5.10s/it]
  9%|▊         | 141/1621 [14:46<2:03:08,  4.99s/it]
  9%|▉         | 142/1621 [14:51<2:05:27,  5.09s/it]
 
0: {'loss': 0.228, 'grad_norm': 0.2833699712446876, 'learning_rate': 9.94e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.09}
0:  9%|▉         | 143/1621 [14:56<2:02:29,  4.97s/it]
  9%|▉         | 144/1621 [15:01<2:01:18,  4.93s/it]
  9%|▉         | 145/1621 [15:06<1:59:54,  4.87s/it]
  9%|▉         | 146/1621 [15:10<1:58:29,  4.82s/it]
  9%|▉         | 147/1621 [15:15<2:00:36,  4.91s/it]
  9%|▉         | 148/1621 [15:20<1:59:48,  4.88s/it]
  9%|▉         | 149/1621 [15:25<1:58:34,  4.83s/it]
  9%|▉         | 150/1621 [15:30<1:59:20,  4.87s/it]
                                                    

  9%|▉         | 150/1621 [15:30<1:59:20,  4.87s/it]
  9%|▉         | 151/1621 [15:35<2:03:59,  5.06s/it]
  9%|▉         | 152/1621 [15:41<2:10:10,  5.32s/it]
  9%|▉         | 153/1621 [15:46<2:05:24,  5.13s/it]
 10%|▉         | 154/1621 [15:51<2:02:58,  5.03s/it]
 10%|▉         | 155/1621 [15:56<2:03:28,  5.05s/it]
 10%|▉         | 156/1621 [16:01<2:01:18,  4.97s/it]
 10%|▉         | 157/1621 [16:06<2:02:57,  5.04s/it]
 10%|▉         | 158/1621 [16:12<2:07:42,  5.24s/it]
 10%|▉         | 159/1621 [16:16<2
0: {'loss': 0.2244, 'grad_norm': 0.2798382710926688, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.1}
0: {'loss': 0.2304, 'grad_norm': 0.2784530895613703, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.1}
0: :04:33,  5.11s/it]
 10%|▉         | 160/1621 [16:21<2:01:43,  5.00s/it]
                                                    

 10%|▉         | 160/1621 [16:21<2:01:43,  5.00s/it]
 10%|▉         | 161/1621 [16:26<2:01:13,  4.98s/it]
 10%|▉         | 162/1621 [16:31<2:01:08,  4.98s/it]
 10%|█         | 163/1621 [16:36<1:59:32,  4.92s/it]
 10%|█         | 164/1621 [16:41<1:57:23,  4.83s/it]
 10%|█         | 165/1621 [16:45<1:57:19,  4.83s/it]
 10%|█         | 166/1621 [16:50<1:56:06,  4.79s/it]
 10%|█         | 167/1621 [16:55<1:59:03,  4.91s/it]
 10%|█         | 168/1621 [17:00<1:57:33,  4.85s/it]
 10%|█         | 169/1621 [17:05<2:00:04,  4.96s/it]
 10%|█         | 170/1621 [17:10<1:58:32,  4.90s/it]
                                                    

 10%|█         | 170/1621 [17:10<1:58:32,  4.90s/it]
 11%|█         | 171/1621 [17:15<1:57:41,  4.87s/it]
 11%|█         | 172/1621 [17:19<1:56:43,  4.83s/it]
 11%|█         | 173/1621 [17:24<1:55:32,  4.79s/it]
 11%|█         
0: {'loss': 0.231, 'grad_norm': 0.27996196551038177, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.11}
0: | 174/1621 [17:29<1:54:47,  4.76s/it]
 11%|█         | 175/1621 [17:34<1:55:19,  4.79s/it]
 11%|█         | 176/1621 [17:38<1:55:23,  4.79s/it]
 11%|█         | 177/1621 [17:43<1:54:37,  4.76s/it]
 11%|█         | 178/1621 [17:48<1:53:59,  4.74s/it]
 11%|█         | 179/1621 [17:53<1:53:36,  4.73s/it]
 11%|█         | 180/1621 [17:57<1:54:59,  4.79s/it]
                                                    

 11%|█         | 180/1621 [17:58<1:54:59,  4.79s/it]
 11%|█         | 181/1621 [18:02<1:54:16,  4.76s/it]
 11%|█         | 182/1621 [18:07<1:53:40,  4.74s/it]
 11%|█▏        | 183/1621 [18:12<1:53:21,  4.73s/it]
 11%|█▏        | 184/1621 [18:16<1:53:21,  4.73s/it]
 11%|█▏        | 185/1621 [18:21<1:52:48,  4.71s/it]
 11%|█▏        | 186/1621 [18:26<1:54:24,  4.78s/it]
 12%|█▏        | 187/1621 [18:31<1:56:56,  4.89s/it]
 12%|█▏        | 188/1621 [18:37<2:00:52,  5.06s/it]
 12%|█▏        | 189/1621 [18:43<2:08:38,  5.39s/it]
 12%|█▏        | 190/1621 [18:48<2
0: {'loss': 0.2245, 'grad_norm': 0.2792720116191664, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.12}
0: {'loss': 0.2339, 'grad_norm': 0.26943311530651054, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.12}
0: :04:39,  5.23s/it]
                                                    

 12%|█▏        | 190/1621 [18:48<2:04:39,  5.23s/it]
 12%|█▏        | 191/1621 [18:53<2:02:38,  5.15s/it]
 12%|█▏        | 192/1621 [18:57<1:59:58,  5.04s/it]
 12%|█▏        | 193/1621 [19:02<1:58:00,  4.96s/it]
 12%|█▏        | 194/1621 [19:07<1:57:40,  4.95s/it]
 12%|█▏        | 195/1621 [19:12<1:55:55,  4.88s/it]
 12%|█▏        | 196/1621 [19:16<1:54:57,  4.84s/it]
 12%|█▏        | 197/1621 [19:21<1:54:23,  4.82s/it]
 12%|█▏        | 198/1621 [19:26<1:53:31,  4.79s/it]
 12%|█▏        | 199/1621 [19:31<1:53:11,  4.78s/it]
 12%|█▏        | 200/1621 [19:36<1:53:26,  4.79s/it]
                                                    

 12%|█▏        | 200/1621 [19:36<1:53:26,  4.79s/it]
 12%|█▏        | 201/1621 [19:40<1:52:24,  4.75s/it]
 12%|█▏        | 202/1621 [19:45<1:51:58,  4.73s/it]
 13%|█▎        | 203/1621 [19:50<1:52:02,  4.74s/it]
 13%|█▎        | 204/1621 [19:54<1:52:
0: {'loss': 0.2227, 'grad_norm': 0.2691123392185592, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.13}
0: 29,  4.76s/it]
 13%|█▎        | 205/1621 [19:59<1:53:14,  4.80s/it]
 13%|█▎        | 206/1621 [20:04<1:52:19,  4.76s/it]
 13%|█▎        | 207/1621 [20:09<1:52:45,  4.78s/it]
 13%|█▎        | 208/1621 [20:14<1:53:06,  4.80s/it]
 13%|█▎        | 209/1621 [20:18<1:52:43,  4.79s/it]
 13%|█▎        | 210/1621 [20:23<1:53:31,  4.83s/it]
                                                    

 13%|█▎        | 210/1621 [20:23<1:53:31,  4.83s/it]
 13%|█▎        | 211/1621 [20:28<1:53:30,  4.83s/it]
 13%|█▎        | 212/1621 [20:33<1:53:11,  4.82s/it]
 13%|█▎        | 213/1621 [20:38<1:52:11,  4.78s/it]
 13%|█▎        | 214/1621 [20:43<1:53:32,  4.84s/it]
 13%|█▎        | 215/1621 [20:48<1:54:54,  4.90s/it]
 13%|█▎        | 216/1621 [20:52<1:53:39,  4.85s/it]
 13%|█▎        | 217/1621 [20:57<1:52:41,  4.82s/it]
 13%|█▎        | 218/1621 [21:02<1:52:06,  4.79s/it]
 14%|█▎        | 219/1621 [21:07<1:51:33,  4.77s/it]
 14%|█▎        | 220/1621 [21:11<1:51:0
0: {'loss': 0.2352, 'grad_norm': 0.3657263884364967, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.14}
0: {'loss': 0.2264, 'grad_norm': 0.3037520130063509, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.14}
0: 8,  4.76s/it]
                                                    

 14%|█▎        | 220/1621 [21:11<1:51:08,  4.76s/it]
 14%|█▎        | 221/1621 [21:16<1:50:44,  4.75s/it]
 14%|█▎        | 222/1621 [21:21<1:52:02,  4.81s/it]
 14%|█▍        | 223/1621 [21:26<1:50:49,  4.76s/it]
 14%|█▍        | 224/1621 [21:30<1:50:47,  4.76s/it]
 14%|█▍        | 225/1621 [21:35<1:50:56,  4.77s/it]
 14%|█▍        | 226/1621 [21:40<1:51:13,  4.78s/it]
 14%|█▍        | 227/1621 [21:45<1:50:01,  4.74s/it]
 14%|█▍        | 228/1621 [21:50<1:50:52,  4.78s/it]
 14%|█▍        | 229/1621 [21:54<1:50:11,  4.75s/it]
 14%|█▍        | 230/1621 [21:59<1:51:07,  4.79s/it]
                                                    

 14%|█▍        | 230/1621 [21:59<1:51:07,  4.79s/it]
 14%|█▍        | 231/1621 [22:04<1:50:29,  4.77s/it]
 14%|█▍        | 232/1621 [22:08<1:49:39,  4.74s/it]
 14%|█▍        | 233/1621 [22:13<1:49:34,  4.74s/it]
 14%|█▍        | 234/1621 [22:18<1:49:09,  
0: {'loss': 0.231, 'grad_norm': 0.2936400072170698, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.15}
0: 4.72s/it]
 14%|█▍        | 235/1621 [22:23<1:50:13,  4.77s/it]
 15%|█▍        | 236/1621 [22:28<1:49:57,  4.76s/it]
 15%|█▍        | 237/1621 [22:32<1:49:10,  4.73s/it]
 15%|█▍        | 238/1621 [22:37<1:49:15,  4.74s/it]
 15%|█▍        | 239/1621 [22:42<1:49:34,  4.76s/it]
 15%|█▍        | 240/1621 [22:46<1:48:47,  4.73s/it]
                                                    

 15%|█▍        | 240/1621 [22:46<1:48:47,  4.73s/it]
 15%|█▍        | 241/1621 [22:51<1:48:11,  4.70s/it]
 15%|█▍        | 242/1621 [22:56<1:47:47,  4.69s/it]
 15%|█▍        | 243/1621 [23:00<1:47:46,  4.69s/it]
 15%|█▌        | 244/1621 [23:05<1:47:33,  4.69s/it]
 15%|█▌        | 245/1621 [23:10<1:48:45,  4.74s/it]
 15%|█▌        | 246/1621 [23:15<1:48:13,  4.72s/it]
 15%|█▌        | 247/1621 [23:21<1:58:08,  5.16s/it]
 15%|█▌        | 248/1621 [23:26<1:55:14,  5.04s/it]
 15%|█▌        | 249/1621 [23:31<1:57:59,  5.16s/it]
 15%|█▌        | 250/1621 [23:36<1:54:34,  5
0: {'loss': 0.225, 'grad_norm': 0.2749950026180422, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.15}
0: {'loss': 0.2254, 'grad_norm': 0.27253893456551315, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.16}
0: .01s/it]
                                                    

 15%|█▌        | 250/1621 [23:36<1:54:34,  5.01s/it]
 15%|█▌        | 251/1621 [23:40<1:52:51,  4.94s/it]
 16%|█▌        | 252/1621 [23:45<1:51:51,  4.90s/it]
 16%|█▌        | 253/1621 [23:50<1:52:12,  4.92s/it]
 16%|█▌        | 254/1621 [23:55<1:50:42,  4.86s/it]
 16%|█▌        | 255/1621 [24:00<1:51:10,  4.88s/it]
 16%|█▌        | 256/1621 [24:05<1:50:46,  4.87s/it]
 16%|█▌        | 257/1621 [24:10<1:50:58,  4.88s/it]
 16%|█▌        | 258/1621 [24:14<1:49:41,  4.83s/it]
 16%|█▌        | 259/1621 [24:19<1:49:52,  4.84s/it]
 16%|█▌        | 260/1621 [24:24<1:49:30,  4.83s/it]
                                                    

 16%|█▌        | 260/1621 [24:24<1:49:30,  4.83s/it]
 16%|█▌        | 261/1621 [24:29<1:48:43,  4.80s/it]
 16%|█▌        | 262/1621 [24:33<1:48:12,  4.78s/it]
 16%|█▌        | 263/1621 [24:39<1:51:03,  4.91s/it]
 16%|█▋        | 264/1621 [24:44<1:50:55,  4.90s
0: {'loss': 0.223, 'grad_norm': 0.28707421840879827, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.17}
0: /it]
 16%|█▋        | 265/1621 [24:49<1:51:38,  4.94s/it]
 16%|█▋        | 266/1621 [24:54<1:53:18,  5.02s/it]
 16%|█▋        | 267/1621 [25:00<1:58:37,  5.26s/it]
 17%|█▋        | 268/1621 [25:04<1:54:50,  5.09s/it]
 17%|█▋        | 269/1621 [25:09<1:52:18,  4.98s/it]
 17%|█▋        | 270/1621 [25:14<1:50:21,  4.90s/it]
                                                    

 17%|█▋        | 270/1621 [25:14<1:50:21,  4.90s/it]
 17%|█▋        | 271/1621 [25:19<1:50:30,  4.91s/it]
 17%|█▋        | 272/1621 [25:24<1:51:59,  4.98s/it]
 17%|█▋        | 273/1621 [25:30<1:57:14,  5.22s/it]
 17%|█▋        | 274/1621 [25:35<1:55:12,  5.13s/it]
 17%|█▋        | 275/1621 [25:39<1:52:27,  5.01s/it]
 17%|█▋        | 276/1621 [25:44<1:53:28,  5.06s/it]
 17%|█▋        | 277/1621 [25:49<1:52:16,  5.01s/it]
 17%|█▋        | 278/1621 [25:54<1:49:54,  4.91s/it]
 17%|█▋        | 279/1621 [25:59<1:49:03,  4.88s/it]
 17%|█▋        | 280/1621 [26:04<1:49:15,  4.89s/
0: {'loss': 0.2273, 'grad_norm': 0.29934907545031625, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.17}
0: {'loss': 0.2251, 'grad_norm': 0.27152873873106187, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.18}
0: it]
                                                    

 17%|█▋        | 280/1621 [26:04<1:49:15,  4.89s/it]
 17%|█▋        | 281/1621 [26:09<1:54:52,  5.14s/it]
 17%|█▋        | 282/1621 [26:14<1:52:43,  5.05s/it]
 17%|█▋        | 283/1621 [26:19<1:51:16,  4.99s/it]
 18%|█▊        | 284/1621 [26:24<1:49:26,  4.91s/it]
 18%|█▊        | 285/1621 [26:29<1:48:12,  4.86s/it]
 18%|█▊        | 286/1621 [26:33<1:46:58,  4.81s/it]
 18%|█▊        | 287/1621 [26:38<1:46:48,  4.80s/it]
 18%|█▊        | 288/1621 [26:43<1:47:36,  4.84s/it]
 18%|█▊        | 289/1621 [26:48<1:48:04,  4.87s/it]
 18%|█▊        | 290/1621 [26:53<1:47:51,  4.86s/it]
                                                    

 18%|█▊        | 290/1621 [26:53<1:47:51,  4.86s/it]
 18%|█▊        | 291/1621 [26:58<1:47:39,  4.86s/it]
 18%|█▊        | 292/1621 [27:03<1:50:40,  5.00s/it]
 18%|█▊        | 293/1621 [27:08<1:48:20,  4.90s/it]
 18%|█▊        | 294/1621 [27:12<1:47:16,  4.85s/it]
0: {'loss': 0.2256, 'grad_norm': 0.2697161270992779, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.19}
0:  18%|█▊        | 295/1621 [27:17<1:48:02,  4.89s/it]
 18%|█▊        | 296/1621 [27:22<1:48:22,  4.91s/it]
 18%|█▊        | 297/1621 [27:27<1:47:10,  4.86s/it]
 18%|█▊        | 298/1621 [27:32<1:46:31,  4.83s/it]
 18%|█▊        | 299/1621 [27:37<1:47:51,  4.89s/it]
 19%|█▊        | 300/1621 [27:42<1:46:42,  4.85s/it]
                                                    

 19%|█▊        | 300/1621 [27:42<1:46:42,  4.85s/it]
 19%|█▊        | 301/1621 [27:46<1:46:00,  4.82s/it]
 19%|█▊        | 302/1621 [27:51<1:45:18,  4.79s/it]
 19%|█▊        | 303/1621 [27:57<1:52:19,  5.11s/it]
 19%|█▉        | 304/1621 [28:02<1:49:50,  5.00s/it]
 19%|█▉        | 305/1621 [28:06<1:48:05,  4.93s/it]
 19%|█▉        | 306/1621 [28:11<1:47:23,  4.90s/it]
 19%|█▉        | 307/1621 [28:16<1:45:56,  4.84s/it]
 19%|█▉        | 308/1621 [28:21<1:45:46,  4.83s/it]
 19%|█▉        | 309/1621 [28:26<1:44:51,  4.80s/it]
 19%|█▉        | 310/1621 [28:30<1:44:15,  4.77s/it]
 
0: {'loss': 0.2215, 'grad_norm': 0.2703094434846044, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.19}
0: {'loss': 0.2252, 'grad_norm': 0.28310536455434837, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.2}
0:                                                    

 19%|█▉        | 310/1621 [28:30<1:44:15,  4.77s/it]
 19%|█▉        | 311/1621 [28:36<1:48:13,  4.96s/it]
 19%|█▉        | 312/1621 [28:41<1:48:06,  4.96s/it]
 19%|█▉        | 313/1621 [28:45<1:46:19,  4.88s/it]
 19%|█▉        | 314/1621 [28:50<1:45:08,  4.83s/it]
 19%|█▉        | 315/1621 [28:55<1:45:27,  4.85s/it]
 19%|█▉        | 316/1621 [29:00<1:44:51,  4.82s/it]
 20%|█▉        | 317/1621 [29:04<1:44:44,  4.82s/it]
 20%|█▉        | 318/1621 [29:10<1:47:42,  4.96s/it]
 20%|█▉        | 319/1621 [29:15<1:47:43,  4.96s/it]
 20%|█▉        | 320/1621 [29:19<1:46:25,  4.91s/it]
                                                    

 20%|█▉        | 320/1621 [29:19<1:46:25,  4.91s/it]
 20%|█▉        | 321/1621 [29:24<1:44:57,  4.84s/it]
 20%|█▉        | 322/1621 [29:29<1:44:11,  4.81s/it]
 20%|█▉        | 323/1621 [29:34<1:43:16,  4.77s/it]
 20%|█▉        | 324/1621 [29:38<1:42:38,  4.75s/it]
 20%|
0: [2025-09-02 19:34:06,837] [INFO] [axolotl.core.trainers.base._save:613] [PID:2134977] [RANK:0] Saving model checkpoint to /lustre/fswork/projects/rech/dgo/udv55np/math/Qwen3-235B-A22B/Qwen2.5-14B_ift/0/checkpoint-325
0: [2025-09-02 19:34:29,509] [INFO] [axolotl.core.trainers.base._save:662] [PID:2134977] [RANK:0] Saving Trainer.data_collator.tokenizer by default as Trainer.processing_class is `None`
0: {'loss': 0.2218, 'grad_norm': 0.2993205202255554, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.2}
0: ██        | 325/1621 [29:44<1:47:14,  4.97s/it]
 20%|██        | 326/1621 [30:30<6:15:14, 17.39s/it]
 20%|██        | 327/1621 [30:36<4:57:55, 13.81s/it]
 20%|██        | 328/1621 [30:40<3:59:57, 11.13s/it]
 20%|██        | 329/1621 [30:45<3:18:22,  9.21s/it]
 20%|██        | 330/1621 [30:50<2:48:55,  7.85s/it]
                                                    

 20%|██        | 330/1621 [30:50<2:48:55,  7.85s/it]
 20%|██        | 331/1621 [30:55<2:28:26,  6.90s/it]
 20%|██        | 332/1621 [30:59<2:14:10,  6.25s/it]
 21%|██        | 333/1621 [31:04<2:05:04,  5.83s/it]
 21%|██        | 334/1621 [31:09<2:01:34,  5.67s/it]
 21%|██        | 335/1621 [31:14<1:56:50,  5.45s/it]
 21%|██        | 336/1621 [31:20<1:56:11,  5.43s/it]
 21%|██        | 337/1621 [31:24<1:51:24,  5.21s/it]
 21%|██        | 338/1621 [31:29<1:48:01,  5.05s/it]
 21%|██        | 339/1621 [31:34<1:47:23,  5.03s/it]
 21%|██        | 340/1621 [31:39<1:44:55,  4.91s/it]
      
0: {'loss': 0.226, 'grad_norm': 0.27889848818452406, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.21}
0: {'loss': 0.2205, 'grad_norm': 0.2827087303201053, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.22}
0:                                               

 21%|██        | 340/1621 [31:39<1:44:55,  4.91s/it]
 21%|██        | 341/1621 [31:45<1:52:00,  5.25s/it]
 21%|██        | 342/1621 [31:50<1:48:55,  5.11s/it]
 21%|██        | 343/1621 [31:55<1:50:00,  5.16s/it]
 21%|██        | 344/1621 [32:00<1:48:47,  5.11s/it]
 21%|██▏       | 345/1621 [32:05<1:47:45,  5.07s/it]
 21%|██▏       | 346/1621 [32:09<1:44:58,  4.94s/it]
 21%|██▏       | 347/1621 [32:14<1:43:21,  4.87s/it]
 21%|██▏       | 348/1621 [32:19<1:43:11,  4.86s/it]
 22%|██▏       | 349/1621 [32:24<1:41:37,  4.79s/it]
 22%|██▏       | 350/1621 [32:28<1:41:48,  4.81s/it]
                                                    

 22%|██▏       | 350/1621 [32:29<1:41:48,  4.81s/it]
 22%|██▏       | 351/1621 [32:33<1:41:34,  4.80s/it]
 22%|██▏       | 352/1621 [32:38<1:40:53,  4.77s/it]
 22%|██▏       | 353/1621 [32:43<1:40:09,  4.74s/it]
 22%|██▏       | 354/1621 [32:48<1:41:13,
0: {'loss': 0.2276, 'grad_norm': 0.2806893131234233, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.22}
0:   4.79s/it]
 22%|██▏       | 355/1621 [32:52<1:40:42,  4.77s/it]
 22%|██▏       | 356/1621 [32:57<1:40:14,  4.75s/it]
 22%|██▏       | 357/1621 [33:02<1:39:51,  4.74s/it]
 22%|██▏       | 358/1621 [33:06<1:39:47,  4.74s/it]
 22%|██▏       | 359/1621 [33:11<1:39:50,  4.75s/it]
 22%|██▏       | 360/1621 [33:16<1:41:07,  4.81s/it]
                                                    

 22%|██▏       | 360/1621 [33:16<1:41:07,  4.81s/it]
 22%|██▏       | 361/1621 [33:21<1:41:13,  4.82s/it]
 22%|██▏       | 362/1621 [33:26<1:41:29,  4.84s/it]
 22%|██▏       | 363/1621 [33:31<1:40:38,  4.80s/it]
 22%|██▏       | 364/1621 [33:35<1:41:05,  4.83s/it]
 23%|██▎       | 365/1621 [33:40<1:40:00,  4.78s/it]
 23%|██▎       | 366/1621 [33:45<1:39:25,  4.75s/it]
 23%|██▎       | 367/1621 [33:51<1:47:26,  5.14s/it]
 23%|██▎       | 368/1621 [33:56<1:46:27,  5.10s/it]
 23%|██▎       | 369/1621 [34:01<1:44:50,  5.02s/it]
 23%|██▎
0: {'loss': 0.2228, 'grad_norm': 0.2891434638471352, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.23}
0: {'loss': 0.2249, 'grad_norm': 0.2937072467224044, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.23}
0:        | 370/1621 [34:06<1:43:29,  4.96s/it]
                                                    

 23%|██▎       | 370/1621 [34:06<1:43:29,  4.96s/it]
 23%|██▎       | 371/1621 [34:10<1:41:40,  4.88s/it]
 23%|██▎       | 372/1621 [34:15<1:40:28,  4.83s/it]
 23%|██▎       | 373/1621 [34:20<1:40:47,  4.85s/it]
 23%|██▎       | 374/1621 [34:25<1:41:22,  4.88s/it]
 23%|██▎       | 375/1621 [34:30<1:40:46,  4.85s/it]
 23%|██▎       | 376/1621 [34:35<1:43:29,  4.99s/it]
 23%|██▎       | 377/1621 [34:40<1:41:56,  4.92s/it]
 23%|██▎       | 378/1621 [34:45<1:42:50,  4.96s/it]
 23%|██▎       | 379/1621 [34:51<1:49:28,  5.29s/it]
 23%|██▎       | 380/1621 [34:56<1:46:14,  5.14s/it]
                                                    

 23%|██▎       | 380/1621 [34:56<1:46:14,  5.14s/it]
 24%|██▎       | 381/1621 [35:00<1:43:24,  5.00s/it]
 24%|██▎       | 382/1621 [35:05<1:42:04,  4.94s/it]
 24%|██▎       | 383/1621 [35:10<1:40:2
0: {'loss': 0.2224, 'grad_norm': 0.28778273745914995, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.24}
0: 4,  4.87s/it]
 24%|██▎       | 384/1621 [35:16<1:47:21,  5.21s/it]
 24%|██▍       | 385/1621 [35:21<1:45:31,  5.12s/it]
 24%|██▍       | 386/1621 [35:26<1:48:57,  5.29s/it]
 24%|██▍       | 387/1621 [35:31<1:44:54,  5.10s/it]
 24%|██▍       | 388/1621 [35:36<1:45:26,  5.13s/it]
 24%|██▍       | 389/1621 [35:41<1:42:28,  4.99s/it]
 24%|██▍       | 390/1621 [35:46<1:41:26,  4.94s/it]
                                                    

 24%|██▍       | 390/1621 [35:46<1:41:26,  4.94s/it]
 24%|██▍       | 391/1621 [35:50<1:40:10,  4.89s/it]
 24%|██▍       | 392/1621 [35:56<1:41:45,  4.97s/it]
 24%|██▍       | 393/1621 [36:00<1:40:15,  4.90s/it]
 24%|██▍       | 394/1621 [36:06<1:47:42,  5.27s/it]
 24%|██▍       | 395/1621 [36:12<1:46:47,  5.23s/it]
 24%|██▍       | 396/1621 [36:16<1:43:31,  5.07s/it]
 24%|██▍       | 397/1621 [36:21<1:41:40,  4.98s/it]
 25%|██▍       | 398/1621 [36:26<1:39:39,  4.89s/it]
 25%|██�
0: {'loss': 0.2161, 'grad_norm': 0.29578423224080963, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.25}
0: {'loss': 0.2226, 'grad_norm': 0.2776599089421953, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.25}
0: ��       | 399/1621 [36:30<1:38:21,  4.83s/it]
 25%|██▍       | 400/1621 [36:36<1:39:47,  4.90s/it]
                                                    

 25%|██▍       | 400/1621 [36:36<1:39:47,  4.90s/it]
 25%|██▍       | 401/1621 [36:40<1:39:31,  4.89s/it]
 25%|██▍       | 402/1621 [36:46<1:41:50,  5.01s/it]
 25%|██▍       | 403/1621 [36:51<1:43:47,  5.11s/it]
 25%|██▍       | 404/1621 [36:56<1:41:13,  4.99s/it]
 25%|██▍       | 405/1621 [37:01<1:42:15,  5.05s/it]
 25%|██▌       | 406/1621 [37:06<1:40:11,  4.95s/it]
 25%|██▌       | 407/1621 [37:10<1:39:04,  4.90s/it]
 25%|██▌       | 408/1621 [37:15<1:38:15,  4.86s/it]
 25%|██▌       | 409/1621 [37:20<1:37:13,  4.81s/it]
 25%|██▌       | 410/1621 [37:25<1:36:42,  4.79s/it]
                                                    

 25%|██▌       | 410/1621 [37:25<1:36:42,  4.79s/it]
 25%|██▌       | 411/1621 [37:29<1:36:19,  4.78s/it]
 25%|██▌       | 412/1621 [37:34<1:36
0: {'loss': 0.2202, 'grad_norm': 0.2851396473633923, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.26}
0: :04,  4.77s/it]
 25%|██▌       | 413/1621 [37:39<1:38:57,  4.92s/it]
 26%|██▌       | 414/1621 [37:44<1:37:37,  4.85s/it]
 26%|██▌       | 415/1621 [37:49<1:36:40,  4.81s/it]
 26%|██▌       | 416/1621 [37:54<1:40:00,  4.98s/it]
 26%|██▌       | 417/1621 [37:59<1:38:12,  4.89s/it]
 26%|██▌       | 418/1621 [38:04<1:36:40,  4.82s/it]
 26%|██▌       | 419/1621 [38:09<1:38:59,  4.94s/it]
 26%|██▌       | 420/1621 [38:13<1:37:37,  4.88s/it]
                                                    

 26%|██▌       | 420/1621 [38:13<1:37:37,  4.88s/it]
 26%|██▌       | 421/1621 [38:18<1:36:14,  4.81s/it]
 26%|██▌       | 422/1621 [38:23<1:35:30,  4.78s/it]
 26%|██▌       | 423/1621 [38:28<1:34:55,  4.75s/it]
 26%|██▌       | 424/1621 [38:33<1:38:07,  4.92s/it]
 26%|██▌       | 425/1621 [38:38<1:37:06,  4.87s/it]
 26%|██▋       | 426/1621 [38:43<1:41:53,  5.12s/it]
 26%|██▋       | 427/1621 [38:48<1:40:17,  5.04s/it]
 26%|█�
0: {'loss': 0.226, 'grad_norm': 0.3027055991780978, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.27}
0: {'loss': 0.2171, 'grad_norm': 0.2775325370268343, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.27}
0: �▋       | 428/1621 [38:53<1:38:18,  4.94s/it]
 26%|██▋       | 429/1621 [38:58<1:36:40,  4.87s/it]
 27%|██▋       | 430/1621 [39:02<1:35:42,  4.82s/it]
                                                    

 27%|██▋       | 430/1621 [39:02<1:35:42,  4.82s/it]
 27%|██▋       | 431/1621 [39:07<1:37:34,  4.92s/it]
 27%|██▋       | 432/1621 [39:12<1:36:44,  4.88s/it]
 27%|██▋       | 433/1621 [39:17<1:37:25,  4.92s/it]
 27%|██▋       | 434/1621 [39:23<1:44:10,  5.27s/it]
 27%|██▋       | 435/1621 [39:28<1:41:00,  5.11s/it]
 27%|██▋       | 436/1621 [39:33<1:39:14,  5.02s/it]
 27%|██▋       | 437/1621 [39:38<1:37:52,  4.96s/it]
 27%|██▋       | 438/1621 [39:43<1:40:38,  5.10s/it]
 27%|██▋       | 439/1621 [39:48<1:38:08,  4.98s/it]
 27%|██▋       | 440/1621 [39:53<1:39:14,  5.04s/it]
                                                    

 27%|██▋       | 440/1621 [39:53<1:39:14,  5.04s/it]
 27%|██▋       | 441/1621 [39:58<1:
0: {'loss': 0.2224, 'grad_norm': 0.28083751378878014, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.28}
0: 37:07,  4.94s/it]
 27%|██▋       | 442/1621 [40:03<1:37:12,  4.95s/it]
 27%|██▋       | 443/1621 [40:07<1:35:41,  4.87s/it]
 27%|██▋       | 444/1621 [40:12<1:34:35,  4.82s/it]
 27%|██▋       | 445/1621 [40:17<1:34:23,  4.82s/it]
 28%|██▊       | 446/1621 [40:23<1:42:49,  5.25s/it]
 28%|██▊       | 447/1621 [40:28<1:39:29,  5.08s/it]
 28%|██▊       | 448/1621 [40:33<1:37:15,  4.97s/it]
 28%|██▊       | 449/1621 [40:37<1:35:56,  4.91s/it]
 28%|██▊       | 450/1621 [40:42<1:35:52,  4.91s/it]
                                                    

 28%|██▊       | 450/1621 [40:42<1:35:52,  4.91s/it]
 28%|██▊       | 451/1621 [40:47<1:36:38,  4.96s/it]
 28%|██▊       | 452/1621 [40:52<1:36:38,  4.96s/it]
 28%|██▊       | 453/1621 [40:57<1:35:08,  4.89s/it]
 28%|██▊       | 454/1621 [41:02<1:34:12,  4.84s/it]
 28%|██▊       | 455/1621 [41:06<1:33:20,  4.80s/it]
 28%|██▊       | 456/1621 [41:11<1:32:59,  4.79s/it]
 28%|█
0: {'loss': 0.2217, 'grad_norm': 0.29695254217700356, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.28}
0: {'loss': 0.2201, 'grad_norm': 0.27273838019805313, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.29}
0: █▊       | 457/1621 [41:17<1:37:21,  5.02s/it]
 28%|██▊       | 458/1621 [41:21<1:35:11,  4.91s/it]
 28%|██▊       | 459/1621 [41:26<1:34:12,  4.86s/it]
 28%|██▊       | 460/1621 [41:31<1:32:54,  4.80s/it]
                                                    

 28%|██▊       | 460/1621 [41:31<1:32:54,  4.80s/it]
 28%|██▊       | 461/1621 [41:35<1:32:10,  4.77s/it]
 29%|██▊       | 462/1621 [41:40<1:32:55,  4.81s/it]
 29%|██▊       | 463/1621 [41:45<1:33:33,  4.85s/it]
 29%|██▊       | 464/1621 [41:50<1:33:54,  4.87s/it]
 29%|██▊       | 465/1621 [41:55<1:33:14,  4.84s/it]
 29%|██▊       | 466/1621 [42:00<1:32:05,  4.78s/it]
 29%|██▉       | 467/1621 [42:05<1:33:47,  4.88s/it]
 29%|██▉       | 468/1621 [42:09<1:32:47,  4.83s/it]
 29%|██▉       | 469/1621 [42:14<1:32:52,  4.84s/it]
 29%|██▉       | 470/1621 [42:19<1:33:00,  4.85s/it]
                                                    

 29%|██▉       | 470/1621 [42:19<
0: {'loss': 0.224, 'grad_norm': 0.29445797771312915, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.3}
0: 1:33:00,  4.85s/it]
 29%|██▉       | 471/1621 [42:25<1:37:01,  5.06s/it]
 29%|██▉       | 472/1621 [42:29<1:34:39,  4.94s/it]
 29%|██▉       | 473/1621 [42:35<1:39:18,  5.19s/it]
 29%|██▉       | 474/1621 [42:40<1:37:07,  5.08s/it]
 29%|██▉       | 475/1621 [42:45<1:34:53,  4.97s/it]
 29%|██▉       | 476/1621 [42:49<1:33:02,  4.88s/it]
 29%|██▉       | 477/1621 [42:54<1:32:32,  4.85s/it]
 29%|██▉       | 478/1621 [42:59<1:31:41,  4.81s/it]
 30%|██▉       | 479/1621 [43:04<1:30:34,  4.76s/it]
 30%|██▉       | 480/1621 [43:08<1:30:11,  4.74s/it]
                                                    

 30%|██▉       | 480/1621 [43:08<1:30:11,  4.74s/it]
 30%|██▉       | 481/1621 [43:13<1:29:45,  4.72s/it]
 30%|██▉       | 482/1621 [43:18<1:29:25,  4.71s/it]
 30%|██▉       | 483/1621 [43:23<1:34:09,  4.96s/it]
 30%|██▉       | 484/1621 [43:28<1:32:47,  4.90s/it]
 30%|██▉       | 485/1621 [43:33<1:33:42,  4.95s/it]
 30%|�
0: {'loss': 0.2164, 'grad_norm': 0.27358498215933197, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.3}
0: ��█▉       | 486/1621 [43:38<1:34:06,  4.97s/it]
 30%|███       | 487/1621 [43:43<1:33:49,  4.96s/it]
 30%|███       | 488/1621 [43:48<1:32:14,  4.88s/it]
 30%|███       | 489/1621 [43:53<1:36:36,  5.12s/it]
 30%|███       | 490/1621 [43:58<1:35:07,  5.05s/it]
                                                    

 30%|███       | 490/1621 [43:58<1:35:07,  5.05s/it]
 30%|███       | 491/1621 [44:03<1:36:21,  5.12s/it]
 30%|███       | 492/1621 [44:09<1:36:55,  5.15s/it]
 30%|███       | 493/1621 [44:14<1:40:00,  5.32s/it]
 30%|███       | 494/1621 [44:19<1:36:20,  5.13s/it]
 31%|███       | 495/1621 [44:24<1:35:47,  5.10s/it]
 31%|███       | 496/1621 [44:29<1:35:16,  5.08s/it]
 31%|███       | 497/1621 [44:34<1:33:39,  5.00s/it]
 31%|███       | 498/1621 [44:39<1:33:44,  5.01s/it]
 31%|███       | 499/1621 [44:44<1:32:40,  4.96s/it]
 31%|███       | 500/1621 [44:49<1:31:23,  4.89s/it]
                                
0: {'loss': 0.2163, 'grad_norm': 0.2699071388397595, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.31}
0: {'loss': 0.2203, 'grad_norm': 0.3025178334899167, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.31}
0:                     

 31%|███       | 500/1621 [44:49<1:31:23,  4.89s/it]
 31%|███       | 501/1621 [44:54<1:32:10,  4.94s/it]
 31%|███       | 502/1621 [44:59<1:32:15,  4.95s/it]
 31%|███       | 503/1621 [45:03<1:31:08,  4.89s/it]
 31%|███       | 504/1621 [45:08<1:30:56,  4.88s/it]
 31%|███       | 505/1621 [45:13<1:30:18,  4.86s/it]
 31%|███       | 506/1621 [45:18<1:29:30,  4.82s/it]
 31%|███▏      | 507/1621 [45:22<1:28:54,  4.79s/it]
 31%|███▏      | 508/1621 [45:27<1:29:52,  4.84s/it]
 31%|███▏      | 509/1621 [45:32<1:29:50,  4.85s/it]
 31%|███▏      | 510/1621 [45:37<1:29:20,  4.83s/it]
                                                    

 31%|███▏      | 510/1621 [45:37<1:29:20,  4.83s/it]
 32%|███▏      | 511/1621 [45:42<1:29:45,  4.85s/it]
 32%|███▏      | 512/1621 [45:47<1:29:00,  4.82s/it]
 32%|███▏      | 513/1621 [45:52<1:32:24,  5.00s/it]
 32%|███▏      | 514/1621 [45:57<1:32:2
0: {'loss': 0.2162, 'grad_norm': 0.2858585805555601, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.32}
0: 6,  5.01s/it]
 32%|███▏      | 515/1621 [46:03<1:35:25,  5.18s/it]
 32%|███▏      | 516/1621 [46:07<1:32:36,  5.03s/it]
 32%|███▏      | 517/1621 [46:12<1:30:54,  4.94s/it]
 32%|███▏      | 518/1621 [46:17<1:29:17,  4.86s/it]
 32%|███▏      | 519/1621 [46:23<1:34:39,  5.15s/it]
 32%|███▏      | 520/1621 [46:27<1:32:11,  5.02s/it]
                                                    

 32%|███▏      | 520/1621 [46:27<1:32:11,  5.02s/it]
 32%|███▏      | 521/1621 [46:32<1:30:34,  4.94s/it]
 32%|███▏      | 522/1621 [46:37<1:30:00,  4.91s/it]
 32%|███▏      | 523/1621 [46:42<1:29:12,  4.88s/it]
 32%|███▏      | 524/1621 [46:48<1:33:54,  5.14s/it]
 32%|███▏      | 525/1621 [46:52<1:31:21,  5.00s/it]
 32%|███▏      | 526/1621 [46:57<1:29:44,  4.92s/it]
 33%|███▎      | 527/1621 [47:02<1:29:19,  4.90s/it]
 33%|███▎      | 528/1621 [47:06<1:28:10,  4.84s/it]
 33%|███▎      | 529/1621 [47:12<
0: {'loss': 0.2196, 'grad_norm': 0.2825619726348337, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.33}
0: {'loss': 0.2215, 'grad_norm': 0.2806598223572053, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.33}
0: 1:29:32,  4.92s/it]
 33%|███▎      | 530/1621 [47:16<1:28:29,  4.87s/it]
                                                    

 33%|███▎      | 530/1621 [47:16<1:28:29,  4.87s/it]
 33%|███▎      | 531/1621 [47:21<1:28:25,  4.87s/it]
 33%|███▎      | 532/1621 [47:26<1:27:42,  4.83s/it]
 33%|███▎      | 533/1621 [47:31<1:30:30,  4.99s/it]
 33%|███▎      | 534/1621 [47:37<1:31:48,  5.07s/it]
 33%|███▎      | 535/1621 [47:41<1:30:32,  5.00s/it]
 33%|███▎      | 536/1621 [47:46<1:28:57,  4.92s/it]
 33%|███▎      | 537/1621 [47:51<1:27:39,  4.85s/it]
 33%|███▎      | 538/1621 [47:56<1:27:35,  4.85s/it]
 33%|███▎      | 539/1621 [48:00<1:26:59,  4.82s/it]
 33%|███▎      | 540/1621 [48:05<1:27:08,  4.84s/it]
                                                    

 33%|███▎      | 540/1621 [48:05<1:27:08,  4.84s/it]
 33%|███▎      | 541/1621 [48:10<1:26:06,  4.78s/it]
 33%|███▎      | 542/1621 [48:15<1
0: {'loss': 0.2184, 'grad_norm': 0.27993941650756166, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.34}
0: :25:35,  4.76s/it]
 33%|███▎      | 543/1621 [48:19<1:25:28,  4.76s/it]
 34%|███▎      | 544/1621 [48:24<1:25:08,  4.74s/it]
 34%|███▎      | 545/1621 [48:29<1:26:02,  4.80s/it]
 34%|███▎      | 546/1621 [48:34<1:26:31,  4.83s/it]
 34%|███▎      | 547/1621 [48:39<1:26:52,  4.85s/it]
 34%|███▍      | 548/1621 [48:44<1:26:15,  4.82s/it]
 34%|███▍      | 549/1621 [48:48<1:26:04,  4.82s/it]
 34%|███▍      | 550/1621 [48:53<1:26:15,  4.83s/it]
                                                    

 34%|███▍      | 550/1621 [48:53<1:26:15,  4.83s/it]
 34%|███▍      | 551/1621 [48:58<1:25:58,  4.82s/it]
 34%|███▍      | 552/1621 [49:03<1:26:37,  4.86s/it]
 34%|███▍      | 553/1621 [49:08<1:25:27,  4.80s/it]
 34%|███▍      | 554/1621 [49:12<1:24:54,  4.77s/it]
 34%|███▍      | 555/1621 [49:17<1:24:07,  4.74s/it]
 34%|███▍      | 556/1621 [49:22<1:24:15,  4.75s/it]
 34%|███▍      | 557/1621 [4
0: {'loss': 0.2188, 'grad_norm': 0.2717601936897116, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.35}
0: {'loss': 0.2228, 'grad_norm': 0.2931548668073716, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.35}
0: 9:27<1:26:01,  4.85s/it]
 34%|███▍      | 558/1621 [49:32<1:25:55,  4.85s/it]
 34%|███▍      | 559/1621 [49:37<1:25:56,  4.86s/it]
 35%|███▍      | 560/1621 [49:42<1:27:39,  4.96s/it]
                                                    

 35%|███▍      | 560/1621 [49:42<1:27:39,  4.96s/it]
 35%|███▍      | 561/1621 [49:47<1:26:40,  4.91s/it]
 35%|███▍      | 562/1621 [49:53<1:33:42,  5.31s/it]
 35%|███▍      | 563/1621 [49:58<1:33:17,  5.29s/it]
 35%|███▍      | 564/1621 [50:03<1:30:03,  5.11s/it]
 35%|███▍      | 565/1621 [50:08<1:28:36,  5.03s/it]
 35%|███▍      | 566/1621 [50:12<1:26:43,  4.93s/it]
 35%|███▍      | 567/1621 [50:18<1:29:34,  5.10s/it]
 35%|███▌      | 568/1621 [50:23<1:27:18,  4.97s/it]
 35%|███▌      | 569/1621 [50:27<1:25:47,  4.89s/it]
 35%|███▌      | 570/1621 [50:32<1:24:39,  4.83s/it]
                                                    

 35%|███▌      | 570/1621 [50
0: {'loss': 0.218, 'grad_norm': 0.2654817797833893, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.36}
0: :32<1:24:39,  4.83s/it]
 35%|███▌      | 571/1621 [50:37<1:25:35,  4.89s/it]
 35%|███▌      | 572/1621 [50:42<1:25:45,  4.90s/it]
 35%|███▌      | 573/1621 [50:47<1:24:37,  4.84s/it]
 35%|███▌      | 574/1621 [50:51<1:23:48,  4.80s/it]
 35%|███▌      | 575/1621 [50:56<1:23:47,  4.81s/it]
 36%|███▌      | 576/1621 [51:01<1:24:19,  4.84s/it]
 36%|███▌      | 577/1621 [51:06<1:24:33,  4.86s/it]
 36%|███▌      | 578/1621 [51:11<1:23:36,  4.81s/it]
 36%|███▌      | 579/1621 [51:15<1:23:06,  4.79s/it]
 36%|███▌      | 580/1621 [51:20<1:22:32,  4.76s/it]
                                                    

 36%|███▌      | 580/1621 [51:20<1:22:32,  4.76s/it]
 36%|███▌      | 581/1621 [51:25<1:22:10,  4.74s/it]
 36%|███▌      | 582/1621 [51:29<1:21:37,  4.71s/it]
 36%|███▌      | 583/1621 [51:35<1:25:53,  4.96s/it]
 36%|███▌      | 584/1621 [51:40<1:26:44,  5.02s/it]
 36%|███▌      | 585/16
0: {'loss': 0.2206, 'grad_norm': 0.2843657162386269, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.36}
0: 21 [51:45<1:25:17,  4.94s/it]
 36%|███▌      | 586/1621 [51:50<1:23:56,  4.87s/it]
 36%|███▌      | 587/1621 [51:54<1:23:29,  4.84s/it]
 36%|███▋      | 588/1621 [51:59<1:22:58,  4.82s/it]
 36%|███▋      | 589/1621 [52:04<1:24:15,  4.90s/it]
 36%|███▋      | 590/1621 [52:10<1:28:35,  5.16s/it]
                                                    

 36%|███▋      | 590/1621 [52:10<1:28:35,  5.16s/it]
 36%|███▋      | 591/1621 [52:15<1:26:31,  5.04s/it]
 37%|███▋      | 592/1621 [52:20<1:28:50,  5.18s/it]
 37%|███▋      | 593/1621 [52:25<1:28:14,  5.15s/it]
 37%|███▋      | 594/1621 [52:30<1:27:53,  5.13s/it]
 37%|███▋      | 595/1621 [52:35<1:26:15,  5.04s/it]
 37%|███▋      | 596/1621 [52:40<1:24:25,  4.94s/it]
 37%|███▋      | 597/1621 [52:45<1:24:07,  4.93s/it]
 37%|███▋      | 598/1621 [52:51<1:28:23,  5.18s/it]
 37%|███▋      | 599/1621 [52:55<1:25:55,  5.04s/it]
 37%|███▋      | 
0: {'loss': 0.2101, 'grad_norm': 0.27242097410054383, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.37}
0: {'loss': 0.2175, 'grad_norm': 0.27369902586673506, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.38}
0: 600/1621 [53:00<1:23:55,  4.93s/it]
                                                    

 37%|███▋      | 600/1621 [53:00<1:23:55,  4.93s/it]
 37%|███▋      | 601/1621 [53:05<1:23:34,  4.92s/it]
 37%|███▋      | 602/1621 [53:10<1:26:38,  5.10s/it]
 37%|███▋      | 603/1621 [53:15<1:24:15,  4.97s/it]
 37%|███▋      | 604/1621 [53:21<1:30:06,  5.32s/it]
 37%|███▋      | 605/1621 [53:26<1:29:23,  5.28s/it]
 37%|███▋      | 606/1621 [53:31<1:26:26,  5.11s/it]
 37%|███▋      | 607/1621 [53:36<1:23:41,  4.95s/it]
 38%|███▊      | 608/1621 [53:41<1:22:55,  4.91s/it]
 38%|███▊      | 609/1621 [53:45<1:21:42,  4.84s/it]
 38%|███▊      | 610/1621 [53:50<1:21:27,  4.83s/it]
                                                    

 38%|███▊      | 610/1621 [53:50<1:21:27,  4.83s/it]
 38%|███▊      | 611/1621 [53:55<1:20:29,  4.78s/it]
 38%|███▊      | 612/1621 [54:00<1:21:05,  4.82s/it]
 38%|███▊      | 6
0: {'loss': 0.2196, 'grad_norm': 0.2717722682759322, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.38}
0: 13/1621 [54:04<1:20:14,  4.78s/it]
 38%|███▊      | 614/1621 [54:09<1:19:55,  4.76s/it]
 38%|███▊      | 615/1621 [54:15<1:26:51,  5.18s/it]
 38%|███▊      | 616/1621 [54:20<1:25:15,  5.09s/it]
 38%|███▊      | 617/1621 [54:25<1:23:04,  4.96s/it]
 38%|███▊      | 618/1621 [54:30<1:23:40,  5.01s/it]
 38%|███▊      | 619/1621 [54:35<1:22:21,  4.93s/it]
 38%|███▊      | 620/1621 [54:39<1:20:55,  4.85s/it]
                                                    

 38%|███▊      | 620/1621 [54:39<1:20:55,  4.85s/it]
 38%|███▊      | 621/1621 [54:44<1:19:55,  4.80s/it]
 38%|███▊      | 622/1621 [54:49<1:19:23,  4.77s/it]
 38%|███▊      | 623/1621 [54:53<1:19:02,  4.75s/it]
 38%|███▊      | 624/1621 [54:58<1:20:18,  4.83s/it]
 39%|███▊      | 625/1621 [55:03<1:19:20,  4.78s/it]
 39%|███▊      | 626/1621 [55:08<1:20:15,  4.84s/it]
 39%|███▊      | 627/1621 [55:13<1:19:29,  4.80s/it]
 39%|███▊   
0: {'loss': 0.2176, 'grad_norm': 0.268329487405915, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.39}
0: {'loss': 0.2193, 'grad_norm': 0.27333488439228737, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.39}
0:    | 628/1621 [55:18<1:23:40,  5.06s/it]
 39%|███▉      | 629/1621 [55:23<1:22:12,  4.97s/it]
 39%|███▉      | 630/1621 [55:28<1:21:34,  4.94s/it]
                                                    

 39%|███▉      | 630/1621 [55:28<1:21:34,  4.94s/it]
 39%|███▉      | 631/1621 [55:33<1:21:46,  4.96s/it]
 39%|███▉      | 632/1621 [55:38<1:20:16,  4.87s/it]
 39%|███▉      | 633/1621 [55:43<1:20:45,  4.90s/it]
 39%|███▉      | 634/1621 [55:48<1:20:58,  4.92s/it]
 39%|███▉      | 635/1621 [55:52<1:19:54,  4.86s/it]
 39%|███▉      | 636/1621 [55:57<1:18:56,  4.81s/it]
 39%|███▉      | 637/1621 [56:02<1:18:56,  4.81s/it]
 39%|███▉      | 638/1621 [56:08<1:24:26,  5.15s/it]
 39%|███▉      | 639/1621 [56:12<1:22:13,  5.02s/it]
 39%|███▉      | 640/1621 [56:17<1:20:30,  4.92s/it]
                                                    

 39%|███▉      | 640/1621 [56:17<1:20:30,  4.92s/it]
 40%|███▉    
0: {'loss': 0.2158, 'grad_norm': 0.27556160207006947, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.4}
0: [2025-09-02 20:01:28,827] [INFO] [axolotl.core.trainers.base._save:613] [PID:2134977] [RANK:0] Saving model checkpoint to /lustre/fswork/projects/rech/dgo/udv55np/math/Qwen3-235B-A22B/Qwen2.5-14B_ift/0/checkpoint-650
0: [2025-09-02 20:01:54,285] [INFO] [axolotl.core.trainers.base._save:662] [PID:2134977] [RANK:0] Saving Trainer.data_collator.tokenizer by default as Trainer.processing_class is `None`
0:   | 641/1621 [56:22<1:19:57,  4.90s/it]
 40%|███▉      | 642/1621 [56:27<1:19:09,  4.85s/it]
 40%|███▉      | 643/1621 [56:32<1:19:15,  4.86s/it]
 40%|███▉      | 644/1621 [56:37<1:19:48,  4.90s/it]
 40%|███▉      | 645/1621 [56:41<1:18:45,  4.84s/it]
 40%|███▉      | 646/1621 [56:46<1:18:09,  4.81s/it]
 40%|███▉      | 647/1621 [56:51<1:17:47,  4.79s/it]
 40%|███▉      | 648/1621 [56:56<1:17:25,  4.77s/it]
 40%|████      | 649/1621 [57:01<1:21:16,  5.02s/it]
 40%|████      | 650/1621 [57:06<1:20:10,  4.95s/it]
                                                    

 40%|████      | 650/1621 [57:06<1:20:10,  4.95s/it]
 40%|████      | 651/1621 [57:56<4:59:26, 18.52s/it]
 40%|████      | 652/1621 [58:02<3:56:36, 14.65s/it]
 40%|████      | 653/1621 [58:07<3:09:42, 11.76s/it]
 40%|████      | 654/1621 [58:11<2:35:22,  9.64s/it]
 40%|████      | 655/1621 [58:16<2:11:15,  8.15s/it]
 40%|███�
0: {'loss': 0.2128, 'grad_norm': 0.2692142059802366, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.41}
0: ��      | 656/1621 [58:21<1:54:23,  7.11s/it]
 41%|████      | 657/1621 [58:26<1:42:34,  6.38s/it]
 41%|████      | 658/1621 [58:31<1:36:38,  6.02s/it]
 41%|████      | 659/1621 [58:36<1:31:13,  5.69s/it]
 41%|████      | 660/1621 [58:41<1:30:11,  5.63s/it]
                                                    

 41%|████      | 660/1621 [58:41<1:30:11,  5.63s/it]
 41%|████      | 661/1621 [58:46<1:26:07,  5.38s/it]
 41%|████      | 662/1621 [58:51<1:22:57,  5.19s/it]
 41%|████      | 663/1621 [58:55<1:20:44,  5.06s/it]
 41%|████      | 664/1621 [59:00<1:19:03,  4.96s/it]
 41%|████      | 665/1621 [59:05<1:17:34,  4.87s/it]
 41%|████      | 666/1621 [59:09<1:16:47,  4.82s/it]
 41%|████      | 667/1621 [59:14<1:16:01,  4.78s/it]
 41%|████      | 668/1621 [59:20<1:19:48,  5.02s/it]
 41%|████▏     | 669/1621 [59:25<1:19:00,  4.98s/it]
 41%|████▏     | 670/1621 [59:29<1:18:08,  4.93s/it]
     
0: {'loss': 0.218, 'grad_norm': 0.2715865676756254, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.41}
0: {'loss': 0.21, 'grad_norm': 0.2775979473941244, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.42}
0:                                                

 41%|████▏     | 670/1621 [59:29<1:18:08,  4.93s/it]
 41%|████▏     | 671/1621 [59:34<1:17:11,  4.88s/it]
 41%|████▏     | 672/1621 [59:39<1:16:47,  4.86s/it]
 42%|████▏     | 673/1621 [59:44<1:16:18,  4.83s/it]
 42%|████▏     | 674/1621 [59:49<1:17:10,  4.89s/it]
 42%|████▏     | 675/1621 [59:54<1:17:20,  4.91s/it]
 42%|████▏     | 676/1621 [59:58<1:16:32,  4.86s/it]
 42%|████▏     | 677/1621 [1:00:04<1:19:44,  5.07s/it]
 42%|████▏     | 678/1621 [1:00:09<1:17:59,  4.96s/it]
 42%|████▏     | 679/1621 [1:00:14<1:17:28,  4.93s/it]
 42%|████▏     | 680/1621 [1:00:19<1:18:21,  5.00s/it]
                                                      

 42%|████▏     | 680/1621 [1:00:19<1:18:21,  5.00s/it]
 42%|████▏     | 681/1621 [1:00:24<1:19:52,  5.10s/it]
 42%|████▏     | 682/1621 [1:00:29<1:19:02,  5.05s/it]
 42%|████▏   
0: {'loss': 0.2163, 'grad_norm': 0.2767364037640067, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.43}
0:   | 683/1621 [1:00:34<1:17:50,  4.98s/it]
 42%|████▏     | 684/1621 [1:00:39<1:16:51,  4.92s/it]
 42%|████▏     | 685/1621 [1:00:43<1:16:08,  4.88s/it]
 42%|████▏     | 686/1621 [1:00:49<1:17:43,  4.99s/it]
 42%|████▏     | 687/1621 [1:00:53<1:16:22,  4.91s/it]
 42%|████▏     | 688/1621 [1:00:58<1:15:15,  4.84s/it]
 43%|████▎     | 689/1621 [1:01:03<1:14:19,  4.79s/it]
 43%|████▎     | 690/1621 [1:01:08<1:17:20,  4.98s/it]
                                                      

 43%|████▎     | 690/1621 [1:01:08<1:17:20,  4.98s/it]
 43%|████▎     | 691/1621 [1:01:13<1:16:10,  4.91s/it]
 43%|████▎     | 692/1621 [1:01:18<1:16:26,  4.94s/it]
 43%|████▎     | 693/1621 [1:01:23<1:17:31,  5.01s/it]
 43%|████▎     | 694/1621 [1:01:28<1:17:42,  5.03s/it]
 43%|████▎     | 695/1621 [1:01:33<1:18:26,  5.08s/it]
 43%|████▎     | 696/1621 [1:01:38<1:16:49,  4.98s/it]
 43%|███�
0: {'loss': 0.2166, 'grad_norm': 0.27600448503199765, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.43}
0: �▎     | 697/1621 [1:01:43<1:15:41,  4.92s/it]
 43%|████▎     | 698/1621 [1:01:48<1:14:51,  4.87s/it]
 43%|████▎     | 699/1621 [1:01:52<1:14:18,  4.84s/it]
 43%|████▎     | 700/1621 [1:01:57<1:14:01,  4.82s/it]
                                                      

 43%|████▎     | 700/1621 [1:01:57<1:14:01,  4.82s/it]
 43%|████▎     | 701/1621 [1:02:02<1:13:37,  4.80s/it]
 43%|████▎     | 702/1621 [1:02:07<1:13:18,  4.79s/it]
 43%|████▎     | 703/1621 [1:02:11<1:12:59,  4.77s/it]
 43%|████▎     | 704/1621 [1:02:16<1:12:36,  4.75s/it]
 43%|████▎     | 705/1621 [1:02:21<1:12:12,  4.73s/it]
 44%|████▎     | 706/1621 [1:02:26<1:12:38,  4.76s/it]
 44%|████▎     | 707/1621 [1:02:30<1:12:30,  4.76s/it]
 44%|████▎     | 708/1621 [1:02:35<1:12:27,  4.76s/it]
 44%|████▎     | 709/1621 [1:02:40<1:14:21,  4.89s/it]
 44%|████▍     | 710/1621 [1:02:45<1:14:01,  4.88s/it]
         
0: {'loss': 0.2181, 'grad_norm': 0.3437401810798713, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.44}
0: {'loss': 0.2182, 'grad_norm': 0.2802402195008618, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.44}
0:                                              

 44%|████▍     | 710/1621 [1:02:45<1:14:01,  4.88s/it]
 44%|████▍     | 711/1621 [1:02:50<1:13:06,  4.82s/it]
 44%|████▍     | 712/1621 [1:02:55<1:12:22,  4.78s/it]
 44%|████▍     | 713/1621 [1:02:59<1:12:32,  4.79s/it]
 44%|████▍     | 714/1621 [1:03:04<1:11:54,  4.76s/it]
 44%|████▍     | 715/1621 [1:03:09<1:11:54,  4.76s/it]
 44%|████▍     | 716/1621 [1:03:15<1:18:56,  5.23s/it]
 44%|████▍     | 717/1621 [1:03:20<1:16:35,  5.08s/it]
 44%|████▍     | 718/1621 [1:03:25<1:14:54,  4.98s/it]
 44%|████▍     | 719/1621 [1:03:29<1:13:43,  4.90s/it]
 44%|████▍     | 720/1621 [1:03:34<1:12:51,  4.85s/it]
                                                      

 44%|████▍     | 720/1621 [1:03:34<1:12:51,  4.85s/it]
 44%|████▍     | 721/1621 [1:03:39<1:12:50,  4.86s/it]
 45%|████▍     | 722/1621 [1:03:44<1:11:43,  4.79s/it]
 45%|██
0: {'loss': 0.2153, 'grad_norm': 0.32129231597397556, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.45}
0: ██▍     | 723/1621 [1:03:48<1:11:30,  4.78s/it]
 45%|████▍     | 724/1621 [1:03:53<1:10:43,  4.73s/it]
 45%|████▍     | 725/1621 [1:03:58<1:11:14,  4.77s/it]
 45%|████▍     | 726/1621 [1:04:03<1:11:05,  4.77s/it]
 45%|████▍     | 727/1621 [1:04:07<1:11:29,  4.80s/it]
 45%|████▍     | 728/1621 [1:04:12<1:11:53,  4.83s/it]
 45%|████▍     | 729/1621 [1:04:17<1:11:07,  4.78s/it]
 45%|████▌     | 730/1621 [1:04:22<1:12:59,  4.92s/it]
                                                      

 45%|████▌     | 730/1621 [1:04:22<1:12:59,  4.92s/it]
 45%|████▌     | 731/1621 [1:04:27<1:12:23,  4.88s/it]
 45%|████▌     | 732/1621 [1:04:33<1:14:50,  5.05s/it]
 45%|████▌     | 733/1621 [1:04:38<1:14:40,  5.05s/it]
 45%|████▌     | 734/1621 [1:04:44<1:18:41,  5.32s/it]
 45%|████▌     | 735/1621 [1:04:48<1:16:10,  5.16s/it]
 45%|████▌     | 736/1621 [1:04:54<1:17:59,  5.29s/it]
 45%
0: {'loss': 0.2121, 'grad_norm': 0.2642878114528468, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.46}
0: |████▌     | 737/1621 [1:05:00<1:21:28,  5.53s/it]
 46%|████▌     | 738/1621 [1:05:05<1:17:52,  5.29s/it]
 46%|████▌     | 739/1621 [1:05:10<1:17:13,  5.25s/it]
 46%|████▌     | 740/1621 [1:05:16<1:19:31,  5.42s/it]
                                                      

 46%|████▌     | 740/1621 [1:05:16<1:19:31,  5.42s/it]
 46%|████▌     | 741/1621 [1:05:21<1:18:24,  5.35s/it]
 46%|████▌     | 742/1621 [1:05:26<1:16:18,  5.21s/it]
 46%|████▌     | 743/1621 [1:05:30<1:13:52,  5.05s/it]
 46%|████▌     | 744/1621 [1:05:35<1:12:34,  4.97s/it]
 46%|████▌     | 745/1621 [1:05:40<1:11:15,  4.88s/it]
 46%|████▌     | 746/1621 [1:05:45<1:10:42,  4.85s/it]
 46%|████▌     | 747/1621 [1:05:50<1:11:29,  4.91s/it]
 46%|████▌     | 748/1621 [1:05:55<1:14:49,  5.14s/it]
 46%|████▌     | 749/1621 [1:06:00<1:12:41,  5.00s/it]
 46%|████▋     | 750/1621 [1:06:05<1:11:17,  4.91s/i
0: {'loss': 0.212, 'grad_norm': 0.28681062888229386, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.46}
0: {'loss': 0.2132, 'grad_norm': 0.27795940147223497, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.47}
0: t]
                                                      

 46%|████▋     | 750/1621 [1:06:05<1:11:17,  4.91s/it]
 46%|████▋     | 751/1621 [1:06:09<1:10:18,  4.85s/it]
 46%|████▋     | 752/1621 [1:06:14<1:10:40,  4.88s/it]
 46%|████▋     | 753/1621 [1:06:19<1:10:35,  4.88s/it]
 47%|████▋     | 754/1621 [1:06:24<1:11:41,  4.96s/it]
 47%|████▋     | 755/1621 [1:06:29<1:11:26,  4.95s/it]
 47%|████▋     | 756/1621 [1:06:34<1:10:25,  4.88s/it]
 47%|████▋     | 757/1621 [1:06:39<1:09:33,  4.83s/it]
 47%|████▋     | 758/1621 [1:06:44<1:09:47,  4.85s/it]
 47%|████▋     | 759/1621 [1:06:49<1:10:01,  4.87s/it]
 47%|████▋     | 760/1621 [1:06:53<1:09:45,  4.86s/it]
                                                      

 47%|████▋     | 760/1621 [1:06:53<1:09:45,  4.86s/it]
 47%|████▋     | 761/1621 [1:06:58<1:09:38,  4.86s/it]
 47%|████▋     | 762/1621 [1:07:03<1:08:27,  4.78s/it]
0: {'loss': 0.2172, 'grad_norm': 0.297594215387472, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.48}
0: 
 47%|████▋     | 763/1621 [1:07:08<1:08:50,  4.81s/it]
 47%|████▋     | 764/1621 [1:07:13<1:08:20,  4.78s/it]
 47%|████▋     | 765/1621 [1:07:17<1:08:14,  4.78s/it]
 47%|████▋     | 766/1621 [1:07:23<1:11:12,  5.00s/it]
 47%|████▋     | 767/1621 [1:07:28<1:10:15,  4.94s/it]
 47%|████▋     | 768/1621 [1:07:33<1:11:39,  5.04s/it]
 47%|████▋     | 769/1621 [1:07:38<1:11:09,  5.01s/it]
 48%|████▊     | 770/1621 [1:07:43<1:10:56,  5.00s/it]
                                                      

 48%|████▊     | 770/1621 [1:07:43<1:10:56,  5.00s/it]
 48%|████▊     | 771/1621 [1:07:48<1:10:48,  5.00s/it]
 48%|████▊     | 772/1621 [1:07:53<1:09:33,  4.92s/it]
 48%|████▊     | 773/1621 [1:07:57<1:08:40,  4.86s/it]
 48%|████▊     | 774/1621 [1:08:03<1:11:30,  5.07s/it]
 48%|████▊     | 775/1621 [1:08:08<1:10:31,  5.00s/it]
 48%|████▊     | 776/1621 [1:08:12<1:09:39,  4.
0: {'loss': 0.2144, 'grad_norm': 0.28602779425213437, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.48}
0: 95s/it]
 48%|████▊     | 777/1621 [1:08:17<1:08:50,  4.89s/it]
 48%|████▊     | 778/1621 [1:08:22<1:07:58,  4.84s/it]
 48%|████▊     | 779/1621 [1:08:27<1:07:45,  4.83s/it]
 48%|████▊     | 780/1621 [1:08:32<1:10:57,  5.06s/it]
                                                      

 48%|████▊     | 780/1621 [1:08:32<1:10:57,  5.06s/it]
 48%|████▊     | 781/1621 [1:08:37<1:09:54,  4.99s/it]
 48%|████▊     | 782/1621 [1:08:43<1:11:07,  5.09s/it]
 48%|████▊     | 783/1621 [1:08:47<1:09:34,  4.98s/it]
 48%|████▊     | 784/1621 [1:08:52<1:08:36,  4.92s/it]
 48%|████▊     | 785/1621 [1:08:57<1:07:59,  4.88s/it]
 48%|████▊     | 786/1621 [1:09:02<1:10:26,  5.06s/it]
 49%|████▊     | 787/1621 [1:09:07<1:09:00,  4.96s/it]
 49%|████▊     | 788/1621 [1:09:12<1:07:41,  4.88s/it]
 49%|████▊     | 789/1621 [1:09:16<1:07:02,  4.83s/it]
 49%|████▊     | 790/1621 [1:09:21<1:06:
0: {'loss': 0.2207, 'grad_norm': 0.27599624793574346, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.49}
0: {'loss': 0.2172, 'grad_norm': 0.26979823876142667, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.49}
0: 10,  4.78s/it]
                                                      

 49%|████▊     | 790/1621 [1:09:21<1:06:10,  4.78s/it]
 49%|████▉     | 791/1621 [1:09:26<1:05:51,  4.76s/it]
 49%|████▉     | 792/1621 [1:09:31<1:06:21,  4.80s/it]
 49%|████▉     | 793/1621 [1:09:35<1:05:57,  4.78s/it]
 49%|████▉     | 794/1621 [1:09:40<1:05:20,  4.74s/it]
 49%|████▉     | 795/1621 [1:09:45<1:05:12,  4.74s/it]
 49%|████▉     | 796/1621 [1:09:50<1:05:19,  4.75s/it]
 49%|████▉     | 797/1621 [1:09:54<1:05:48,  4.79s/it]
 49%|████▉     | 798/1621 [1:10:00<1:07:00,  4.88s/it]
 49%|████▉     | 799/1621 [1:10:04<1:06:33,  4.86s/it]
 49%|████▉     | 800/1621 [1:10:09<1:05:51,  4.81s/it]
                                                      

 49%|████▉     | 800/1621 [1:10:09<1:05:51,  4.81s/it]
 49%|████▉     | 801/1621 [1:10:14<1:05:46,  4.81s/it]
 49%|████▉     | 802/1621 [1:10:19<1:05:04
0: {'loss': 0.2217, 'grad_norm': 0.29824386294830224, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.5}
0: ,  4.77s/it]
 50%|████▉     | 803/1621 [1:10:23<1:04:48,  4.75s/it]
 50%|████▉     | 804/1621 [1:10:28<1:04:31,  4.74s/it]
 50%|████▉     | 805/1621 [1:10:33<1:04:26,  4.74s/it]
 50%|████▉     | 806/1621 [1:10:38<1:05:44,  4.84s/it]
 50%|████▉     | 807/1621 [1:10:43<1:05:30,  4.83s/it]
 50%|████▉     | 808/1621 [1:10:47<1:05:25,  4.83s/it]
 50%|████▉     | 809/1621 [1:10:52<1:05:15,  4.82s/it]
 50%|████▉     | 810/1621 [1:10:57<1:04:50,  4.80s/it]
                                                      

 50%|████▉     | 810/1621 [1:10:57<1:04:50,  4.80s/it]
 50%|█████     | 811/1621 [1:11:02<1:06:10,  4.90s/it]
 50%|█████     | 812/1621 [1:11:07<1:06:16,  4.92s/it]
 50%|█████     | 813/1621 [1:11:12<1:05:49,  4.89s/it]
 50%|█████     | 814/1621 [1:11:17<1:04:55,  4.83s/it]
 50%|█████     | 815/1621 [1:11:21<1:04:23,  4.79s/it]
 50%|█████     | 816/1621 [1:11:26<
0: {'loss': 0.2133, 'grad_norm': 0.26092437055105083, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.51}
0: 1:04:56,  4.84s/it]
 50%|█████     | 817/1621 [1:11:32<1:07:24,  5.03s/it]
 50%|█████     | 818/1621 [1:11:37<1:07:57,  5.08s/it]
 51%|█████     | 819/1621 [1:11:42<1:07:14,  5.03s/it]
 51%|█████     | 820/1621 [1:11:47<1:05:46,  4.93s/it]
                                                      

 51%|█████     | 820/1621 [1:11:47<1:05:46,  4.93s/it]
 51%|█████     | 821/1621 [1:11:51<1:05:12,  4.89s/it]
 51%|█████     | 822/1621 [1:11:56<1:05:34,  4.92s/it]
 51%|█████     | 823/1621 [1:12:02<1:07:41,  5.09s/it]
 51%|█████     | 824/1621 [1:12:07<1:06:10,  4.98s/it]
 51%|█████     | 825/1621 [1:12:11<1:05:42,  4.95s/it]
 51%|█████     | 826/1621 [1:12:16<1:04:53,  4.90s/it]
 51%|█████     | 827/1621 [1:12:21<1:04:15,  4.86s/it]
 51%|█████     | 828/1621 [1:12:26<1:03:27,  4.80s/it]
 51%|█████     | 829/1621 [1:12:31<1:05:55,  4.99s/it]
 51%|█████     | 830/1621 [1
0: {'loss': 0.2158, 'grad_norm': 0.26693057475276766, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.51}
0: {'loss': 0.2121, 'grad_norm': 0.27091788132485156, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.52}
0: :12:36<1:06:32,  5.05s/it]
                                                      

 51%|█████     | 830/1621 [1:12:36<1:06:32,  5.05s/it]
 51%|█████▏    | 831/1621 [1:12:41<1:05:22,  4.97s/it]
 51%|█████▏    | 832/1621 [1:12:46<1:05:15,  4.96s/it]
 51%|█████▏    | 833/1621 [1:12:51<1:04:33,  4.92s/it]
 51%|█████▏    | 834/1621 [1:12:56<1:06:56,  5.10s/it]
 52%|█████▏    | 835/1621 [1:13:02<1:08:24,  5.22s/it]
 52%|█████▏    | 836/1621 [1:13:07<1:09:48,  5.34s/it]
 52%|█████▏    | 837/1621 [1:13:12<1:08:07,  5.21s/it]
 52%|█████▏    | 838/1621 [1:13:17<1:05:50,  5.05s/it]
 52%|█████▏    | 839/1621 [1:13:22<1:05:20,  5.01s/it]
 52%|█████▏    | 840/1621 [1:13:27<1:05:29,  5.03s/it]
                                                      

 52%|█████▏    | 840/1621 [1:13:27<1:05:29,  5.03s/it]
 52%|█████▏    | 841/1621 [1:13:32<1:04:19,  4.95s/it]
 52%|███�
0: {'loss': 0.2139, 'grad_norm': 0.2710407718400621, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.52}
0: �█▏    | 842/1621 [1:13:37<1:04:39,  4.98s/it]
 52%|█████▏    | 843/1621 [1:13:41<1:03:28,  4.90s/it]
 52%|█████▏    | 844/1621 [1:13:46<1:03:45,  4.92s/it]
 52%|█████▏    | 845/1621 [1:13:51<1:02:56,  4.87s/it]
 52%|█████▏    | 846/1621 [1:13:56<1:02:12,  4.82s/it]
 52%|█████▏    | 847/1621 [1:14:01<1:02:51,  4.87s/it]
 52%|█████▏    | 848/1621 [1:14:06<1:01:57,  4.81s/it]
 52%|█████▏    | 849/1621 [1:14:10<1:01:38,  4.79s/it]
 52%|█████▏    | 850/1621 [1:14:15<1:01:09,  4.76s/it]
                                                      

 52%|█████▏    | 850/1621 [1:14:15<1:01:09,  4.76s/it]
 52%|█████▏    | 851/1621 [1:14:20<1:01:00,  4.75s/it]
 53%|█████▎    | 852/1621 [1:14:24<1:00:45,  4.74s/it]
 53%|█████▎    | 853/1621 [1:14:29<1:00:52,  4.76s/it]
 53%|█████▎    | 854/1621 [1:14:34<1:00:33,  4.74s/it]
 53%|█████▎    | 855/1621 [1:14:39
0: {'loss': 0.215, 'grad_norm': 0.26826181484921574, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.53}
0: <1:00:18,  4.72s/it]
 53%|█████▎    | 856/1621 [1:14:43<59:57,  4.70s/it]  
 53%|█████▎    | 857/1621 [1:14:48<1:00:13,  4.73s/it]
 53%|█████▎    | 858/1621 [1:14:53<1:00:54,  4.79s/it]
 53%|█████▎    | 859/1621 [1:14:58<1:01:15,  4.82s/it]
 53%|█████▎    | 860/1621 [1:15:03<1:00:56,  4.80s/it]
                                                      

 53%|█████▎    | 860/1621 [1:15:03<1:00:56,  4.80s/it]
 53%|█████▎    | 861/1621 [1:15:07<1:00:53,  4.81s/it]
 53%|█████▎    | 862/1621 [1:15:13<1:02:00,  4.90s/it]
 53%|█████▎    | 863/1621 [1:15:17<1:01:14,  4.85s/it]
 53%|█████▎    | 864/1621 [1:15:22<1:00:51,  4.82s/it]
 53%|█████▎    | 865/1621 [1:15:27<1:01:35,  4.89s/it]
 53%|█████▎    | 866/1621 [1:15:32<1:00:50,  4.84s/it]
 53%|█████▎    | 867/1621 [1:15:37<1:00:14,  4.79s/it]
 54%|█████▎    | 868/1621 [1:15:41<59:43,  4.76s/it]  
 54%|█�
0: {'loss': 0.2139, 'grad_norm': 0.2852708208541685, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.54}
0: {'loss': 0.2151, 'grad_norm': 0.28550830834224616, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.54}
0: ��███▎    | 869/1621 [1:15:46<59:47,  4.77s/it]
 54%|█████▎    | 870/1621 [1:15:51<1:01:12,  4.89s/it]
                                                      

 54%|█████▎    | 870/1621 [1:15:51<1:01:12,  4.89s/it]
 54%|█████▎    | 871/1621 [1:15:56<1:00:21,  4.83s/it]
 54%|█████▍    | 872/1621 [1:16:01<59:39,  4.78s/it]  
 54%|█████▍    | 873/1621 [1:16:05<59:39,  4.79s/it]
 54%|█████▍    | 874/1621 [1:16:10<1:00:18,  4.84s/it]
 54%|█████▍    | 875/1621 [1:16:16<1:03:16,  5.09s/it]
 54%|█████▍    | 876/1621 [1:16:21<1:01:48,  4.98s/it]
 54%|█████▍    | 877/1621 [1:16:25<1:00:49,  4.91s/it]
 54%|█████▍    | 878/1621 [1:16:30<1:00:14,  4.87s/it]
 54%|█████▍    | 879/1621 [1:16:35<59:44,  4.83s/it]  
 54%|█████▍    | 880/1621 [1:16:40<59:11,  4.79s/it]
                                                    

 54%|█████▍    | 880/1621 [1:16:40<59:11,  4.7
0: {'loss': 0.2192, 'grad_norm': 0.2804935280560918, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.55}
0: 9s/it]
 54%|█████▍    | 881/1621 [1:16:44<59:14,  4.80s/it]
 54%|█████▍    | 882/1621 [1:16:49<58:58,  4.79s/it]
 54%|█████▍    | 883/1621 [1:16:54<58:17,  4.74s/it]
 55%|█████▍    | 884/1621 [1:16:59<58:07,  4.73s/it]
 55%|█████▍    | 885/1621 [1:17:04<1:01:35,  5.02s/it]
 55%|█████▍    | 886/1621 [1:17:09<1:00:15,  4.92s/it]
 55%|█████▍    | 887/1621 [1:17:14<1:02:01,  5.07s/it]
 55%|█████▍    | 888/1621 [1:17:19<1:00:51,  4.98s/it]
 55%|█████▍    | 889/1621 [1:17:24<59:45,  4.90s/it]  
 55%|█████▍    | 890/1621 [1:17:29<58:57,  4.84s/it]
                                                    

 55%|█████▍    | 890/1621 [1:17:29<58:57,  4.84s/it]
 55%|█████▍    | 891/1621 [1:17:35<1:02:59,  5.18s/it]
 55%|█████▌    | 892/1621 [1:17:40<1:03:56,  5.26s/it]
 55%|█████▌    | 893/1621 [1:17:45<1:02:07,  5.12s/it]
 55%|█████▌    | 894/1621
0: {'loss': 0.2126, 'grad_norm': 0.2750084705232265, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.56}
0:  [1:17:50<1:01:53,  5.11s/it]
 55%|█████▌    | 895/1621 [1:17:55<1:03:00,  5.21s/it]
 55%|█████▌    | 896/1621 [1:18:00<1:01:04,  5.05s/it]
 55%|█████▌    | 897/1621 [1:18:05<1:01:02,  5.06s/it]
 55%|█████▌    | 898/1621 [1:18:10<59:52,  4.97s/it]  
 55%|█████▌    | 899/1621 [1:18:15<59:42,  4.96s/it]
 56%|█████▌    | 900/1621 [1:18:19<58:36,  4.88s/it]
                                                    

 56%|█████▌    | 900/1621 [1:18:19<58:36,  4.88s/it]
 56%|█████▌    | 901/1621 [1:18:24<57:40,  4.81s/it]
 56%|█████▌    | 902/1621 [1:18:29<57:10,  4.77s/it]
 56%|█████▌    | 903/1621 [1:18:34<56:54,  4.76s/it]
 56%|█████▌    | 904/1621 [1:18:38<56:28,  4.73s/it]
 56%|█████▌    | 905/1621 [1:18:44<59:56,  5.02s/it]
 56%|█████▌    | 906/1621 [1:18:49<59:04,  4.96s/it]
 56%|█████▌    | 907/1621 [1:18:53<58:22,  4.90s/it]
 56%|█████�
0: {'loss': 0.2147, 'grad_norm': 0.3443521809729604, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.56}
0: {'loss': 0.2137, 'grad_norm': 0.27078378526006086, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.57}
0: �    | 908/1621 [1:18:58<57:27,  4.84s/it]
 56%|█████▌    | 909/1621 [1:19:03<57:09,  4.82s/it]
 56%|█████▌    | 910/1621 [1:19:09<59:59,  5.06s/it]
                                                    

 56%|█████▌    | 910/1621 [1:19:09<59:59,  5.06s/it]
 56%|█████▌    | 911/1621 [1:19:14<59:47,  5.05s/it]
 56%|█████▋    | 912/1621 [1:19:18<59:03,  5.00s/it]
 56%|█████▋    | 913/1621 [1:19:23<57:41,  4.89s/it]
 56%|█████▋    | 914/1621 [1:19:28<56:49,  4.82s/it]
 56%|█████▋    | 915/1621 [1:19:33<57:46,  4.91s/it]
 57%|█████▋    | 916/1621 [1:19:38<59:06,  5.03s/it]
 57%|█████▋    | 917/1621 [1:19:43<57:45,  4.92s/it]
 57%|█████▋    | 918/1621 [1:19:48<56:50,  4.85s/it]
 57%|█████▋    | 919/1621 [1:19:52<56:21,  4.82s/it]
 57%|█████▋    | 920/1621 [1:19:57<56:55,  4.87s/it]
                                                    

 57%|█████▋    |
0: {'loss': 0.2147, 'grad_norm': 0.31227860783495803, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.57}
0:  920/1621 [1:19:57<56:55,  4.87s/it]
 57%|█████▋    | 921/1621 [1:20:02<56:13,  4.82s/it]
 57%|█████▋    | 922/1621 [1:20:07<56:18,  4.83s/it]
 57%|█████▋    | 923/1621 [1:20:12<55:55,  4.81s/it]
 57%|█████▋    | 924/1621 [1:20:16<55:51,  4.81s/it]
 57%|█████▋    | 925/1621 [1:20:21<55:29,  4.78s/it]
 57%|█████▋    | 926/1621 [1:20:26<55:23,  4.78s/it]
 57%|█████▋    | 927/1621 [1:20:31<55:01,  4.76s/it]
 57%|█████▋    | 928/1621 [1:20:35<54:46,  4.74s/it]
 57%|█████▋    | 929/1621 [1:20:41<56:43,  4.92s/it]
 57%|█████▋    | 930/1621 [1:20:47<1:01:18,  5.32s/it]
                                                      

 57%|█████▋    | 930/1621 [1:20:47<1:01:18,  5.32s/it]
 57%|█████▋    | 931/1621 [1:20:52<1:00:00,  5.22s/it]
 57%|█████▋    | 932/1621 [1:20:57<58:17,  5.08s/it]  
 58%|█████▊    | 933/1621 [1:21:01<56:53,  4.96s/it]
 58%|██�
0: {'loss': 0.2184, 'grad_norm': 0.25947362686664, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.58}
0: �██▊    | 934/1621 [1:21:06<55:49,  4.88s/it]
 58%|█████▊    | 935/1621 [1:21:11<55:13,  4.83s/it]
 58%|█████▊    | 936/1621 [1:21:16<55:14,  4.84s/it]
 58%|█████▊    | 937/1621 [1:21:20<55:23,  4.86s/it]
 58%|█████▊    | 938/1621 [1:21:25<55:05,  4.84s/it]
 58%|█████▊    | 939/1621 [1:21:30<54:32,  4.80s/it]
 58%|█████▊    | 940/1621 [1:21:36<57:50,  5.10s/it]
                                                    

 58%|█████▊    | 940/1621 [1:21:36<57:50,  5.10s/it]
 58%|█████▊    | 941/1621 [1:21:41<57:11,  5.05s/it]
 58%|█████▊    | 942/1621 [1:21:46<57:45,  5.10s/it]
 58%|█████▊    | 943/1621 [1:21:51<57:45,  5.11s/it]
 58%|█████▊    | 944/1621 [1:21:56<56:35,  5.02s/it]
 58%|█████▊    | 945/1621 [1:22:01<55:18,  4.91s/it]
 58%|█████▊    | 946/1621 [1:22:05<54:59,  4.89s/it]
 58%|█████▊    | 947/1621 [1:22:10<55:18,  4.92s/it]
 58%|█
0: {'loss': 0.2102, 'grad_norm': 0.3142685936794017, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.59}
0: {'loss': 0.2198, 'grad_norm': 0.29152014419018757, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.59}
0: ████▊    | 948/1621 [1:22:15<54:35,  4.87s/it]
 59%|█████▊    | 949/1621 [1:22:20<54:45,  4.89s/it]
 59%|█████▊    | 950/1621 [1:22:25<55:06,  4.93s/it]
                                                    

 59%|█████▊    | 950/1621 [1:22:25<55:06,  4.93s/it]
 59%|█████▊    | 951/1621 [1:22:31<58:40,  5.25s/it]
 59%|█████▊    | 952/1621 [1:22:36<57:28,  5.15s/it]
 59%|█████▉    | 953/1621 [1:22:41<58:29,  5.25s/it]
 59%|█████▉    | 954/1621 [1:22:46<56:46,  5.11s/it]
 59%|█████▉    | 955/1621 [1:22:51<55:42,  5.02s/it]
 59%|█████▉    | 956/1621 [1:22:56<54:28,  4.91s/it]
 59%|█████▉    | 957/1621 [1:23:01<55:41,  5.03s/it]
 59%|█████▉    | 958/1621 [1:23:06<56:48,  5.14s/it]
 59%|█████▉    | 959/1621 [1:23:12<58:42,  5.32s/it]
 59%|█████▉    | 960/1621 [1:23:17<56:52,  5.16s/it]
                                                    

 59%|███
0: {'loss': 0.2175, 'grad_norm': 0.28132541400191075, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.6}
0: ██▉    | 960/1621 [1:23:17<56:52,  5.16s/it]
 59%|█████▉    | 961/1621 [1:23:22<55:30,  5.05s/it]
 59%|█████▉    | 962/1621 [1:23:27<55:10,  5.02s/it]
 59%|█████▉    | 963/1621 [1:23:31<54:19,  4.95s/it]
 59%|█████▉    | 964/1621 [1:23:36<53:41,  4.90s/it]
 60%|█████▉    | 965/1621 [1:23:41<53:11,  4.86s/it]
 60%|█████▉    | 966/1621 [1:23:46<52:29,  4.81s/it]
 60%|█████▉    | 967/1621 [1:23:50<51:54,  4.76s/it]
 60%|█████▉    | 968/1621 [1:23:55<52:18,  4.81s/it]
 60%|█████▉    | 969/1621 [1:24:00<52:50,  4.86s/it]
 60%|█████▉    | 970/1621 [1:24:05<52:50,  4.87s/it]
                                                    

 60%|█████▉    | 970/1621 [1:24:05<52:50,  4.87s/it]
 60%|█████▉    | 971/1621 [1:24:10<53:44,  4.96s/it]
 60%|█████▉    | 972/1621 [1:24:15<53:01,  4.90s/it]
 60%|██████    | 973/1621 [1:24:20<52:22,  4.85s/it]
 60%|█�
0: [2025-09-02 20:28:52,563] [INFO] [axolotl.core.trainers.base._save:613] [PID:2134977] [RANK:0] Saving model checkpoint to /lustre/fswork/projects/rech/dgo/udv55np/math/Qwen3-235B-A22B/Qwen2.5-14B_ift/0/checkpoint-975
0: [2025-09-02 20:29:15,164] [INFO] [axolotl.core.trainers.base._save:662] [PID:2134977] [RANK:0] Saving Trainer.data_collator.tokenizer by default as Trainer.processing_class is `None`
0: {'loss': 0.2136, 'grad_norm': 0.27352537796541654, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.6}
0: ��████    | 974/1621 [1:24:25<52:18,  4.85s/it]
 60%|██████    | 975/1621 [1:24:30<52:06,  4.84s/it]
 60%|██████    | 976/1621 [1:25:16<3:06:42, 17.37s/it]
 60%|██████    | 977/1621 [1:25:21<2:26:43, 13.67s/it]
 60%|██████    | 978/1621 [1:25:26<1:57:50, 11.00s/it]
 60%|██████    | 979/1621 [1:25:31<1:37:29,  9.11s/it]
 60%|██████    | 980/1621 [1:25:35<1:23:19,  7.80s/it]
                                                      

 60%|██████    | 980/1621 [1:25:35<1:23:19,  7.80s/it]
 61%|██████    | 981/1621 [1:25:40<1:13:33,  6.90s/it]
 61%|██████    | 982/1621 [1:25:45<1:06:28,  6.24s/it]
 61%|██████    | 983/1621 [1:25:50<1:04:08,  6.03s/it]
 61%|██████    | 984/1621 [1:25:55<59:58,  5.65s/it]  
 61%|██████    | 985/1621 [1:26:00<57:07,  5.39s/it]
 61%|██████    | 986/1621 [1:26:05<54:50,  5.18s/it]
 61%|██████    | 987/1621 [1:26:09<
0: {'loss': 0.2176, 'grad_norm': 0.2789264204643204, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.61}
0: 53:25,  5.06s/it]
 61%|██████    | 988/1621 [1:26:14<52:22,  4.96s/it]
 61%|██████    | 989/1621 [1:26:20<53:35,  5.09s/it]
 61%|██████    | 990/1621 [1:26:24<52:29,  4.99s/it]
                                                    

 61%|██████    | 990/1621 [1:26:24<52:29,  4.99s/it]
 61%|██████    | 991/1621 [1:26:29<51:43,  4.93s/it]
 61%|██████    | 992/1621 [1:26:35<53:42,  5.12s/it]
 61%|██████▏   | 993/1621 [1:26:39<52:30,  5.02s/it]
 61%|██████▏   | 994/1621 [1:26:44<51:25,  4.92s/it]
 61%|██████▏   | 995/1621 [1:26:49<51:13,  4.91s/it]
 61%|██████▏   | 996/1621 [1:26:54<52:44,  5.06s/it]
 62%|██████▏   | 997/1621 [1:27:00<53:05,  5.11s/it]
 62%|██████▏   | 998/1621 [1:27:06<56:20,  5.43s/it]
 62%|██████▏   | 999/1621 [1:27:11<54:31,  5.26s/it]
 62%|██████▏   | 1000/1621 [1:27:15<52:34,  5.08s/it]
                         
0: {'loss': 0.2141, 'grad_norm': 0.2810459322487379, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.62}
0: {'loss': 0.2124, 'grad_norm': 0.28919272828442305, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.62}
0:                             

 62%|██████▏   | 1000/1621 [1:27:15<52:34,  5.08s/it]
 62%|██████▏   | 1001/1621 [1:27:20<51:53,  5.02s/it]
 62%|██████▏   | 1002/1621 [1:27:25<50:58,  4.94s/it]
 62%|██████▏   | 1003/1621 [1:27:30<50:05,  4.86s/it]
 62%|██████▏   | 1004/1621 [1:27:34<49:16,  4.79s/it]
 62%|██████▏   | 1005/1621 [1:27:40<52:54,  5.15s/it]
 62%|██████▏   | 1006/1621 [1:27:46<53:37,  5.23s/it]
 62%|██████▏   | 1007/1621 [1:27:52<55:23,  5.41s/it]
 62%|██████▏   | 1008/1621 [1:27:57<54:46,  5.36s/it]
 62%|██████▏   | 1009/1621 [1:28:02<55:30,  5.44s/it]
 62%|██████▏   | 1010/1621 [1:28:07<53:01,  5.21s/it]
                                                     

 62%|██████▏   | 1010/1621 [1:28:07<53:01,  5.21s/it]
 62%|██████▏   | 1011/1621 [1:28:12<51:35,  5.07s/it]
 62%|██████▏   | 1012/1621 [1:28:17<50:41
0: {'loss': 0.2135, 'grad_norm': 0.27980806989132895, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.63}
0: ,  4.99s/it]
 62%|██████▏   | 1013/1621 [1:28:21<49:59,  4.93s/it]
 63%|██████▎   | 1014/1621 [1:28:26<49:10,  4.86s/it]
 63%|██████▎   | 1015/1621 [1:28:31<48:49,  4.83s/it]
 63%|██████▎   | 1016/1621 [1:28:36<48:30,  4.81s/it]
 63%|██████▎   | 1017/1621 [1:28:40<48:10,  4.79s/it]
 63%|██████▎   | 1018/1621 [1:28:45<48:20,  4.81s/it]
 63%|██████▎   | 1019/1621 [1:28:50<49:02,  4.89s/it]
 63%|██████▎   | 1020/1621 [1:28:55<48:16,  4.82s/it]
                                                     

 63%|██████▎   | 1020/1621 [1:28:55<48:16,  4.82s/it]
 63%|██████▎   | 1021/1621 [1:29:00<47:51,  4.79s/it]
 63%|██████▎   | 1022/1621 [1:29:04<47:49,  4.79s/it]
 63%|██████▎   | 1023/1621 [1:29:09<47:21,  4.75s/it]
 63%|██████▎   | 1024/1621 [1:29:15<49:10,  4.94s/it]
 63%|██████▎   | 1025/1621 [1:29:19<48:24,  4.87s/it]
 63%
0: {'loss': 0.2148, 'grad_norm': 0.29552995472458155, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.64}
0: |██████▎   | 1026/1621 [1:29:24<47:51,  4.83s/it]
 63%|██████▎   | 1027/1621 [1:29:29<47:20,  4.78s/it]
 63%|██████▎   | 1028/1621 [1:29:34<49:21,  4.99s/it]
 63%|██████▎   | 1029/1621 [1:29:39<48:22,  4.90s/it]
 64%|██████▎   | 1030/1621 [1:29:43<47:31,  4.83s/it]
                                                     

 64%|██████▎   | 1030/1621 [1:29:43<47:31,  4.83s/it]
 64%|██████▎   | 1031/1621 [1:29:48<47:03,  4.79s/it]
 64%|██████▎   | 1032/1621 [1:29:53<46:47,  4.77s/it]
 64%|██████▎   | 1033/1621 [1:29:58<46:46,  4.77s/it]
 64%|██████▍   | 1034/1621 [1:30:02<46:28,  4.75s/it]
 64%|██████▍   | 1035/1621 [1:30:07<46:21,  4.75s/it]
 64%|██████▍   | 1036/1621 [1:30:12<45:56,  4.71s/it]
 64%|██████▍   | 1037/1621 [1:30:17<46:25,  4.77s/it]
 64%|██████▍   | 1038/1621 [1:30:21<45:56,  4.73s/it]
 64%|█████�
0: {'loss': 0.2114, 'grad_norm': 0.29673487038816576, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.64}
0: {'loss': 0.2084, 'grad_norm': 0.26864545919062477, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.65}
0: ��▍   | 1039/1621 [1:30:26<47:07,  4.86s/it]
 64%|██████▍   | 1040/1621 [1:30:31<46:35,  4.81s/it]
                                                     

 64%|██████▍   | 1040/1621 [1:30:31<46:35,  4.81s/it]
 64%|██████▍   | 1041/1621 [1:30:36<46:04,  4.77s/it]
 64%|██████▍   | 1042/1621 [1:30:40<45:42,  4.74s/it]
 64%|██████▍   | 1043/1621 [1:30:45<46:04,  4.78s/it]
 64%|██████▍   | 1044/1621 [1:30:50<45:43,  4.76s/it]
 64%|██████▍   | 1045/1621 [1:30:55<47:03,  4.90s/it]
 65%|██████▍   | 1046/1621 [1:31:01<48:18,  5.04s/it]
 65%|██████▍   | 1047/1621 [1:31:06<47:46,  4.99s/it]
 65%|██████▍   | 1048/1621 [1:31:11<48:02,  5.03s/it]
 65%|██████▍   | 1049/1621 [1:31:17<50:26,  5.29s/it]
 65%|██████▍   | 1050/1621 [1:31:21<48:52,  5.14s/it]
                                                     

 65%|██████▍   | 1050/1621 [1:31:21<4
0: {'loss': 0.2142, 'grad_norm': 0.26666640684697085, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.65}
0: 8:52,  5.14s/it]
 65%|██████▍   | 1051/1621 [1:31:26<48:22,  5.09s/it]
 65%|██████▍   | 1052/1621 [1:31:31<47:16,  4.99s/it]
 65%|██████▍   | 1053/1621 [1:31:36<47:14,  4.99s/it]
 65%|██████▌   | 1054/1621 [1:31:42<49:17,  5.22s/it]
 65%|██████▌   | 1055/1621 [1:31:47<47:49,  5.07s/it]
 65%|██████▌   | 1056/1621 [1:31:52<47:48,  5.08s/it]
 65%|██████▌   | 1057/1621 [1:31:56<46:44,  4.97s/it]
 65%|██████▌   | 1058/1621 [1:32:01<46:19,  4.94s/it]
 65%|██████▌   | 1059/1621 [1:32:06<45:52,  4.90s/it]
 65%|██████▌   | 1060/1621 [1:32:11<46:39,  4.99s/it]
                                                     

 65%|██████▌   | 1060/1621 [1:32:11<46:39,  4.99s/it]
 65%|██████▌   | 1061/1621 [1:32:16<46:03,  4.93s/it]
 66%|██████▌   | 1062/1621 [1:32:21<45:18,  4.86s/it]
 66%|██████▌   | 1063/1621 [1:32:26<45:11,  4.86s/it]
0: {'loss': 0.21, 'grad_norm': 0.28093123469034137, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.66}
0:  66%|██████▌   | 1064/1621 [1:32:30<44:38,  4.81s/it]
 66%|██████▌   | 1065/1621 [1:32:35<44:12,  4.77s/it]
 66%|██████▌   | 1066/1621 [1:32:40<43:44,  4.73s/it]
 66%|██████▌   | 1067/1621 [1:32:44<43:47,  4.74s/it]
 66%|██████▌   | 1068/1621 [1:32:49<43:29,  4.72s/it]
 66%|██████▌   | 1069/1621 [1:32:54<43:22,  4.72s/it]
 66%|██████▌   | 1070/1621 [1:32:58<43:11,  4.70s/it]
                                                     

 66%|██████▌   | 1070/1621 [1:32:58<43:11,  4.70s/it]
 66%|██████▌   | 1071/1621 [1:33:03<43:19,  4.73s/it]
 66%|██████▌   | 1072/1621 [1:33:08<43:13,  4.72s/it]
 66%|██████▌   | 1073/1621 [1:33:13<42:55,  4.70s/it]
 66%|██████▋   | 1074/1621 [1:33:17<42:34,  4.67s/it]
 66%|██████▋   | 1075/1621 [1:33:22<42:59,  4.73s/it]
 66%|██████▋   | 1076/1621 [1:33:28<45:19,  4.99s/it]
 66%|████
0: {'loss': 0.2119, 'grad_norm': 0.2831457906289921, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.67}
0: ██▋   | 1077/1621 [1:33:32<44:32,  4.91s/it]
 67%|██████▋   | 1078/1621 [1:33:37<44:12,  4.88s/it]
 67%|██████▋   | 1079/1621 [1:33:42<44:56,  4.98s/it]
 67%|██████▋   | 1080/1621 [1:33:47<44:58,  4.99s/it]
                                                     

 67%|██████▋   | 1080/1621 [1:33:47<44:58,  4.99s/it]
 67%|██████▋   | 1081/1621 [1:33:52<44:23,  4.93s/it]
 67%|██████▋   | 1082/1621 [1:33:57<43:46,  4.87s/it]
 67%|██████▋   | 1083/1621 [1:34:02<43:24,  4.84s/it]
 67%|██████▋   | 1084/1621 [1:34:08<46:58,  5.25s/it]
 67%|██████▋   | 1085/1621 [1:34:13<45:23,  5.08s/it]
 67%|██████▋   | 1086/1621 [1:34:18<45:36,  5.11s/it]
 67%|██████▋   | 1087/1621 [1:34:22<44:22,  4.99s/it]
 67%|██████▋   | 1088/1621 [1:34:27<43:37,  4.91s/it]
 67%|██████▋   | 1089/1621 [1:34:32<43:01,  4.85s/it]
 67%|██████▋   | 109
0: {'loss': 0.2105, 'grad_norm': 0.2612962366533585, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.67}
0: {'loss': 0.208, 'grad_norm': 0.26706625813917995, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.68}
0: 0/1621 [1:34:37<42:36,  4.82s/it]
                                                     

 67%|██████▋   | 1090/1621 [1:34:37<42:36,  4.82s/it]
 67%|██████▋   | 1091/1621 [1:34:42<43:38,  4.94s/it]
 67%|██████▋   | 1092/1621 [1:34:47<45:22,  5.15s/it]
 67%|██████▋   | 1093/1621 [1:34:52<43:58,  5.00s/it]
 67%|██████▋   | 1094/1621 [1:34:57<43:30,  4.95s/it]
 68%|██████▊   | 1095/1621 [1:35:02<43:37,  4.98s/it]
 68%|██████▊   | 1096/1621 [1:35:08<45:54,  5.25s/it]
 68%|██████▊   | 1097/1621 [1:35:14<47:16,  5.41s/it]
 68%|██████▊   | 1098/1621 [1:35:19<46:21,  5.32s/it]
 68%|██████▊   | 1099/1621 [1:35:24<45:09,  5.19s/it]
 68%|██████▊   | 1100/1621 [1:35:28<44:10,  5.09s/it]
                                                     

 68%|██████▊   | 1100/1621 [1:35:29<44:10,  5.09s/it]
 68%|██████▊   | 1101/1621 [1:35:35<46:56,  5.42s/
0: {'loss': 0.2117, 'grad_norm': 0.3142948099091601, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.68}
0: it]
 68%|██████▊   | 1102/1621 [1:35:40<46:07,  5.33s/it]
 68%|██████▊   | 1103/1621 [1:35:45<45:36,  5.28s/it]
 68%|██████▊   | 1104/1621 [1:35:50<44:30,  5.17s/it]
 68%|██████▊   | 1105/1621 [1:35:56<46:28,  5.40s/it]
 68%|██████▊   | 1106/1621 [1:36:01<44:54,  5.23s/it]
 68%|██████▊   | 1107/1621 [1:36:05<43:40,  5.10s/it]
 68%|██████▊   | 1108/1621 [1:36:10<42:33,  4.98s/it]
 68%|██████▊   | 1109/1621 [1:36:15<42:05,  4.93s/it]
 68%|██████▊   | 1110/1621 [1:36:20<41:27,  4.87s/it]
                                                     

 68%|██████▊   | 1110/1621 [1:36:20<41:27,  4.87s/it]
 69%|██████▊   | 1111/1621 [1:36:25<41:40,  4.90s/it]
 69%|██████▊   | 1112/1621 [1:36:30<41:22,  4.88s/it]
 69%|██████▊   | 1113/1621 [1:36:34<41:05,  4.85s/it]
 69%|██████▊   | 1114/1621 [1:36:39<40:58,  4.85s/it]
 69%|██�
0: {'loss': 0.2134, 'grad_norm': 0.2646577996823958, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.69}
0: �███▉   | 1115/1621 [1:36:44<40:26,  4.80s/it]
 69%|██████▉   | 1116/1621 [1:36:49<40:10,  4.77s/it]
 69%|██████▉   | 1117/1621 [1:36:54<41:00,  4.88s/it]
 69%|██████▉   | 1118/1621 [1:36:58<40:21,  4.81s/it]
 69%|██████▉   | 1119/1621 [1:37:03<40:17,  4.82s/it]
 69%|██████▉   | 1120/1621 [1:37:08<39:50,  4.77s/it]
                                                     

 69%|██████▉   | 1120/1621 [1:37:08<39:50,  4.77s/it]
 69%|██████▉   | 1121/1621 [1:37:13<40:22,  4.84s/it]
 69%|██████▉   | 1122/1621 [1:37:17<39:51,  4.79s/it]
 69%|██████▉   | 1123/1621 [1:37:23<42:23,  5.11s/it]
 69%|██████▉   | 1124/1621 [1:37:28<41:13,  4.98s/it]
 69%|██████▉   | 1125/1621 [1:37:33<40:32,  4.90s/it]
 69%|██████▉   | 1126/1621 [1:37:37<39:48,  4.83s/it]
 70%|██████▉   | 1127/1621 [1:37:42<39:21,  4.78s/it]
 70%|██████▉   |
0: {'loss': 0.2106, 'grad_norm': 0.27369103318579074, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.7}
0:  1128/1621 [1:37:47<39:11,  4.77s/it]
 70%|██████▉   | 1129/1621 [1:37:52<40:11,  4.90s/it]
 70%|██████▉   | 1130/1621 [1:37:57<40:38,  4.97s/it]
                                                     

 70%|██████▉   | 1130/1621 [1:37:57<40:38,  4.97s/it]
 70%|██████▉   | 1131/1621 [1:38:02<40:24,  4.95s/it]
 70%|██████▉   | 1132/1621 [1:38:07<39:44,  4.88s/it]
 70%|██████▉   | 1133/1621 [1:38:11<39:20,  4.84s/it]
 70%|██████▉   | 1134/1621 [1:38:16<39:09,  4.82s/it]
 70%|███████   | 1135/1621 [1:38:21<39:11,  4.84s/it]
 70%|███████   | 1136/1621 [1:38:26<38:55,  4.82s/it]
 70%|███████   | 1137/1621 [1:38:31<39:47,  4.93s/it]
 70%|███████   | 1138/1621 [1:38:36<39:28,  4.90s/it]
 70%|███████   | 1139/1621 [1:38:41<39:34,  4.93s/it]
 70%|███████   | 1140/1621 [1:38:46<39:08,  4.88s/it]
                                               
0: {'loss': 0.2075, 'grad_norm': 0.26705269442253254, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.7}
0: {'loss': 0.2098, 'grad_norm': 0.279247722684754, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.71}
0:       

 70%|███████   | 1140/1621 [1:38:46<39:08,  4.88s/it]
 70%|███████   | 1141/1621 [1:38:50<38:32,  4.82s/it]
 70%|███████   | 1142/1621 [1:38:55<38:05,  4.77s/it]
 71%|███████   | 1143/1621 [1:39:01<40:43,  5.11s/it]
 71%|███████   | 1144/1621 [1:39:06<39:28,  4.97s/it]
 71%|███████   | 1145/1621 [1:39:10<39:01,  4.92s/it]
 71%|███████   | 1146/1621 [1:39:16<40:20,  5.10s/it]
 71%|███████   | 1147/1621 [1:39:21<39:31,  5.00s/it]
 71%|███████   | 1148/1621 [1:39:26<41:01,  5.20s/it]
 71%|███████   | 1149/1621 [1:39:31<39:55,  5.08s/it]
 71%|███████   | 1150/1621 [1:39:36<39:55,  5.09s/it]
                                                     

 71%|███████   | 1150/1621 [1:39:36<39:55,  5.09s/it]
 71%|███████   | 1151/1621 [1:39:41<39:46,  5.08s/it]
 71%|███████   | 1152/1621 [1:39:47<41:37,  5.33s/it]
 71%|█�
0: {'loss': 0.2114, 'grad_norm': 0.28121457876267647, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.72}
0: ��█████   | 1153/1621 [1:39:52<40:02,  5.13s/it]
 71%|███████   | 1154/1621 [1:39:57<39:00,  5.01s/it]
 71%|███████▏  | 1155/1621 [1:40:02<39:03,  5.03s/it]
 71%|███████▏  | 1156/1621 [1:40:06<38:07,  4.92s/it]
 71%|███████▏  | 1157/1621 [1:40:11<37:21,  4.83s/it]
 71%|███████▏  | 1158/1621 [1:40:16<37:14,  4.83s/it]
 71%|███████▏  | 1159/1621 [1:40:21<36:59,  4.80s/it]
 72%|███████▏  | 1160/1621 [1:40:25<36:32,  4.76s/it]
                                                     

 72%|███████▏  | 1160/1621 [1:40:25<36:32,  4.76s/it]
 72%|███████▏  | 1161/1621 [1:40:30<36:18,  4.74s/it]
 72%|███████▏  | 1162/1621 [1:40:35<36:04,  4.72s/it]
 72%|███████▏  | 1163/1621 [1:40:39<36:23,  4.77s/it]
 72%|███████▏  | 1164/1621 [1:40:44<36:24,  4.78s/it]
 72%|███████▏  | 1165/1621 [1:40:49<36:07,  4.75s/it]
 7
0: {'loss': 0.2152, 'grad_norm': 0.26571684410130864, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.72}
0: 2%|███████▏  | 1166/1621 [1:40:54<36:14,  4.78s/it]
 72%|███████▏  | 1167/1621 [1:40:58<35:58,  4.76s/it]
 72%|███████▏  | 1168/1621 [1:41:03<35:47,  4.74s/it]
 72%|███████▏  | 1169/1621 [1:41:08<35:53,  4.76s/it]
 72%|███████▏  | 1170/1621 [1:41:13<36:18,  4.83s/it]
                                                     

 72%|███████▏  | 1170/1621 [1:41:13<36:18,  4.83s/it]
 72%|███████▏  | 1171/1621 [1:41:18<36:05,  4.81s/it]
 72%|███████▏  | 1172/1621 [1:41:22<35:47,  4.78s/it]
 72%|███████▏  | 1173/1621 [1:41:27<35:31,  4.76s/it]
 72%|███████▏  | 1174/1621 [1:41:32<35:31,  4.77s/it]
 72%|███████▏  | 1175/1621 [1:41:37<35:11,  4.74s/it]
 73%|███████▎  | 1176/1621 [1:41:41<35:15,  4.75s/it]
 73%|███████▎  | 1177/1621 [1:41:46<35:29,  4.80s/it]
 73%|███████▎  | 1178/1621 [1:41:51<35:33,  4
0: {'loss': 0.2104, 'grad_norm': 0.28659441188197854, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.73}
0: .82s/it]
 73%|███████▎  | 1179/1621 [1:41:56<35:50,  4.86s/it]
 73%|███████▎  | 1180/1621 [1:42:01<35:15,  4.80s/it]
                                                     

 73%|███████▎  | 1180/1621 [1:42:01<35:15,  4.80s/it]
 73%|███████▎  | 1181/1621 [1:42:05<34:57,  4.77s/it]
 73%|███████▎  | 1182/1621 [1:42:10<34:42,  4.74s/it]
 73%|███████▎  | 1183/1621 [1:42:15<35:18,  4.84s/it]
 73%|███████▎  | 1184/1621 [1:42:20<34:54,  4.79s/it]
 73%|███████▎  | 1185/1621 [1:42:25<34:41,  4.77s/it]
 73%|███████▎  | 1186/1621 [1:42:29<34:24,  4.75s/it]
 73%|███████▎  | 1187/1621 [1:42:34<34:09,  4.72s/it]
 73%|███████▎  | 1188/1621 [1:42:39<34:13,  4.74s/it]
 73%|███████▎  | 1189/1621 [1:42:43<33:57,  4.72s/it]
 73%|███████▎  | 1190/1621 [1:42:49<34:57,  4.87s/it]
                                                  
0: {'loss': 0.2125, 'grad_norm': 0.2754546966001141, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.73}
0: {'loss': 0.2078, 'grad_norm': 0.2850579163343314, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.74}
0:    

 73%|███████▎  | 1190/1621 [1:42:49<34:57,  4.87s/it]
 73%|███████▎  | 1191/1621 [1:42:53<34:31,  4.82s/it]
 74%|███████▎  | 1192/1621 [1:42:58<34:10,  4.78s/it]
 74%|███████▎  | 1193/1621 [1:43:03<34:31,  4.84s/it]
 74%|███████▎  | 1194/1621 [1:43:08<34:36,  4.86s/it]
 74%|███████▎  | 1195/1621 [1:43:13<34:13,  4.82s/it]
 74%|███████▍  | 1196/1621 [1:43:18<34:20,  4.85s/it]
 74%|███████▍  | 1197/1621 [1:43:22<34:00,  4.81s/it]
 74%|███████▍  | 1198/1621 [1:43:27<33:50,  4.80s/it]
 74%|███████▍  | 1199/1621 [1:43:32<35:02,  4.98s/it]
 74%|███████▍  | 1200/1621 [1:43:37<34:23,  4.90s/it]
                                                     

 74%|███████▍  | 1200/1621 [1:43:37<34:23,  4.90s/it]
 74%|███████▍  | 1201/1621 [1:43:42<33:51,  4.84s/it]
 74%|███████▍  | 1202/1621 [1:43:47<33
0: {'loss': 0.2153, 'grad_norm': 0.27762596072148293, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.75}
0: :37,  4.81s/it]
 74%|███████▍  | 1203/1621 [1:43:52<34:02,  4.89s/it]
 74%|███████▍  | 1204/1621 [1:43:56<33:30,  4.82s/it]
 74%|███████▍  | 1205/1621 [1:44:01<33:06,  4.77s/it]
 74%|███████▍  | 1206/1621 [1:44:06<33:00,  4.77s/it]
 74%|███████▍  | 1207/1621 [1:44:11<32:51,  4.76s/it]
 75%|███████▍  | 1208/1621 [1:44:16<33:56,  4.93s/it]
 75%|███████▍  | 1209/1621 [1:44:21<34:11,  4.98s/it]
 75%|███████▍  | 1210/1621 [1:44:26<33:52,  4.95s/it]
                                                     

 75%|███████▍  | 1210/1621 [1:44:26<33:52,  4.95s/it]
 75%|███████▍  | 1211/1621 [1:44:31<33:20,  4.88s/it]
 75%|███████▍  | 1212/1621 [1:44:35<32:47,  4.81s/it]
 75%|███████▍  | 1213/1621 [1:44:40<33:17,  4.90s/it]
 75%|███████▍  | 1214/1621 [1:44:45<32:45,  4.83s/it]
 75%|███████▍  | 1215/1621 
0: {'loss': 0.2115, 'grad_norm': 0.2740032175831026, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.75}
0: [1:44:50<33:53,  5.01s/it]
 75%|███████▌  | 1216/1621 [1:44:55<33:02,  4.89s/it]
 75%|███████▌  | 1217/1621 [1:45:00<33:07,  4.92s/it]
 75%|███████▌  | 1218/1621 [1:45:05<33:24,  4.97s/it]
 75%|███████▌  | 1219/1621 [1:45:11<34:16,  5.12s/it]
 75%|███████▌  | 1220/1621 [1:45:16<34:07,  5.11s/it]
                                                     

 75%|███████▌  | 1220/1621 [1:45:16<34:07,  5.11s/it]
 75%|███████▌  | 1221/1621 [1:45:20<33:09,  4.97s/it]
 75%|███████▌  | 1222/1621 [1:45:25<32:29,  4.89s/it]
 75%|███████▌  | 1223/1621 [1:45:30<32:08,  4.85s/it]
 76%|███████▌  | 1224/1621 [1:45:35<33:32,  5.07s/it]
 76%|███████▌  | 1225/1621 [1:45:40<32:42,  4.96s/it]
 76%|███████▌  | 1226/1621 [1:45:45<32:31,  4.94s/it]
 76%|███████▌  | 1227/1621 [1:45:50<32:17,  4.92s/it]
 76%|███████▌  |
0: {'loss': 0.2137, 'grad_norm': 1.012604656340779, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.76}
0:  1228/1621 [1:45:55<32:08,  4.91s/it]
 76%|███████▌  | 1229/1621 [1:46:00<31:54,  4.88s/it]
 76%|███████▌  | 1230/1621 [1:46:04<31:22,  4.82s/it]
                                                     

 76%|███████▌  | 1230/1621 [1:46:04<31:22,  4.82s/it]
 76%|███████▌  | 1231/1621 [1:46:09<31:15,  4.81s/it]
 76%|███████▌  | 1232/1621 [1:46:14<31:40,  4.89s/it]
 76%|███████▌  | 1233/1621 [1:46:19<31:14,  4.83s/it]
 76%|███████▌  | 1234/1621 [1:46:23<30:58,  4.80s/it]
 76%|███████▌  | 1235/1621 [1:46:28<30:59,  4.82s/it]
 76%|███████▌  | 1236/1621 [1:46:33<30:37,  4.77s/it]
 76%|███████▋  | 1237/1621 [1:46:38<30:20,  4.74s/it]
 76%|███████▋  | 1238/1621 [1:46:42<30:15,  4.74s/it]
 76%|███████▋  | 1239/1621 [1:46:47<30:03,  4.72s/it]
 76%|███████▋  | 1240/1621 [1:46:52<29:48,  4.69s/it]
                     
0: {'loss': 0.2143, 'grad_norm': 0.2846388443797818, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.76}
0: {'loss': 0.2055, 'grad_norm': 0.26029718751210856, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.77}
0:                                 

 76%|███████▋  | 1240/1621 [1:46:52<29:48,  4.69s/it]
 77%|███████▋  | 1241/1621 [1:46:57<31:09,  4.92s/it]
 77%|███████▋  | 1242/1621 [1:47:02<31:27,  4.98s/it]
 77%|███████▋  | 1243/1621 [1:47:07<31:00,  4.92s/it]
 77%|███████▋  | 1244/1621 [1:47:12<31:05,  4.95s/it]
 77%|███████▋  | 1245/1621 [1:47:17<31:52,  5.09s/it]
 77%|███████▋  | 1246/1621 [1:47:22<31:09,  4.98s/it]
 77%|███████▋  | 1247/1621 [1:47:27<30:43,  4.93s/it]
 77%|███████▋  | 1248/1621 [1:47:32<30:08,  4.85s/it]
 77%|███████▋  | 1249/1621 [1:47:36<30:01,  4.84s/it]
 77%|███████▋  | 1250/1621 [1:47:41<29:49,  4.82s/it]
                                                     

 77%|███████▋  | 1250/1621 [1:47:41<29:49,  4.82s/it]
 77%|███████▋  | 1251/1621 [1:47:46<29:38,  4.81s/it]
 77%|██████�
0: {'loss': 0.2112, 'grad_norm': 0.35354670964811946, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.78}
0: �▋  | 1252/1621 [1:47:51<29:20,  4.77s/it]
 77%|███████▋  | 1253/1621 [1:47:56<29:51,  4.87s/it]
 77%|███████▋  | 1254/1621 [1:48:01<29:53,  4.89s/it]
 77%|███████▋  | 1255/1621 [1:48:05<29:31,  4.84s/it]
 77%|███████▋  | 1256/1621 [1:48:10<29:11,  4.80s/it]
 78%|███████▊  | 1257/1621 [1:48:15<29:12,  4.81s/it]
 78%|███████▊  | 1258/1621 [1:48:20<28:59,  4.79s/it]
 78%|███████▊  | 1259/1621 [1:48:25<29:38,  4.91s/it]
 78%|███████▊  | 1260/1621 [1:48:30<29:55,  4.97s/it]
                                                     

 78%|███████▊  | 1260/1621 [1:48:30<29:55,  4.97s/it]
 78%|███████▊  | 1261/1621 [1:48:35<29:29,  4.91s/it]
 78%|███████▊  | 1262/1621 [1:48:40<29:48,  4.98s/it]
 78%|███████▊  | 1263/1621 [1:48:45<29:46,  4.99s/it]
 78%|███████▊  | 1264/1621 [1:48:50<29:17,  4.92s/it]
 78%|███
0: {'loss': 0.2116, 'grad_norm': 0.26423705530912817, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.78}
0: ████▊  | 1265/1621 [1:48:55<29:36,  4.99s/it]
 78%|███████▊  | 1266/1621 [1:49:00<29:00,  4.90s/it]
 78%|███████▊  | 1267/1621 [1:49:04<28:34,  4.84s/it]
 78%|███████▊  | 1268/1621 [1:49:09<28:44,  4.88s/it]
 78%|███████▊  | 1269/1621 [1:49:14<28:21,  4.83s/it]
 78%|███████▊  | 1270/1621 [1:49:19<28:04,  4.80s/it]
                                                     

 78%|███████▊  | 1270/1621 [1:49:19<28:04,  4.80s/it]
 78%|███████▊  | 1271/1621 [1:49:23<27:52,  4.78s/it]
 78%|███████▊  | 1272/1621 [1:49:28<27:38,  4.75s/it]
 79%|███████▊  | 1273/1621 [1:49:33<28:08,  4.85s/it]
 79%|███████▊  | 1274/1621 [1:49:38<27:44,  4.80s/it]
 79%|███████▊  | 1275/1621 [1:49:43<27:29,  4.77s/it]
 79%|███████▊  | 1276/1621 [1:49:47<27:29,  4.78s/it]
 79%|███████▉  | 1277/1621 [1:49:52<27:15,  4.75s/it]
 79
0: {'loss': 0.211, 'grad_norm': 0.2708639256375303, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.79}
0: %|███████▉  | 1278/1621 [1:49:57<26:58,  4.72s/it]
 79%|███████▉  | 1279/1621 [1:50:02<27:08,  4.76s/it]
 79%|███████▉  | 1280/1621 [1:50:07<27:17,  4.80s/it]
                                                     

 79%|███████▉  | 1280/1621 [1:50:07<27:17,  4.80s/it]
 79%|███████▉  | 1281/1621 [1:50:11<26:55,  4.75s/it]
 79%|███████▉  | 1282/1621 [1:50:16<26:42,  4.73s/it]
 79%|███████▉  | 1283/1621 [1:50:21<26:33,  4.71s/it]
 79%|███████▉  | 1284/1621 [1:50:25<26:24,  4.70s/it]
 79%|███████▉  | 1285/1621 [1:50:30<26:37,  4.75s/it]
 79%|███████▉  | 1286/1621 [1:50:35<26:23,  4.73s/it]
 79%|███████▉  | 1287/1621 [1:50:39<26:13,  4.71s/it]
 79%|███████▉  | 1288/1621 [1:50:44<26:11,  4.72s/it]
 80%|███████▉  | 1289/1621 [1:50:49<26:39,  4.82s/it]
 80%|███████▉  | 1290/1621 [1:50:54<27:01,  4.
0: {'loss': 0.2084, 'grad_norm': 0.25641950625482923, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.8}
0: {'loss': 0.2098, 'grad_norm': 0.2683510657084983, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.8}
0: [2025-09-02 20:56:07,755] [INFO] [axolotl.core.trainers.base._save:613] [PID:2134977] [RANK:0] Saving model checkpoint to /lustre/fswork/projects/rech/dgo/udv55np/math/Qwen3-235B-A22B/Qwen2.5-14B_ift/0/checkpoint-1300
0: [2025-09-02 20:56:31,010] [INFO] [axolotl.core.trainers.base._save:662] [PID:2134977] [RANK:0] Saving Trainer.data_collator.tokenizer by default as Trainer.processing_class is `None`
0: 90s/it]
                                                     

 80%|███████▉  | 1290/1621 [1:50:54<27:01,  4.90s/it]
 80%|███████▉  | 1291/1621 [1:50:59<26:39,  4.85s/it]
 80%|███████▉  | 1292/1621 [1:51:04<26:21,  4.81s/it]
 80%|███████▉  | 1293/1621 [1:51:09<26:25,  4.83s/it]
 80%|███████▉  | 1294/1621 [1:51:13<26:10,  4.80s/it]
 80%|███████▉  | 1295/1621 [1:51:18<26:07,  4.81s/it]
 80%|███████▉  | 1296/1621 [1:51:24<28:07,  5.19s/it]
 80%|████████  | 1297/1621 [1:51:30<28:45,  5.33s/it]
 80%|████████  | 1298/1621 [1:51:35<28:27,  5.29s/it]
 80%|████████  | 1299/1621 [1:51:40<27:22,  5.10s/it]
 80%|████████  | 1300/1621 [1:51:45<27:20,  5.11s/it]
                                                     

 80%|████████  | 1300/1621 [1:51:45<27:20,  5.11s/it]
 80%|████████  | 1301/1621 [1:52:32<1:33:48, 17.59
0: {'loss': 0.2155, 'grad_norm': 0.2922501736371335, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.81}
0: s/it]
 80%|████████  | 1302/1621 [1:52:36<1:12:54, 13.71s/it]
 80%|████████  | 1303/1621 [1:52:41<58:41, 11.07s/it]  
 80%|████████  | 1304/1621 [1:52:46<48:27,  9.17s/it]
 81%|████████  | 1305/1621 [1:52:51<41:11,  7.82s/it]
 81%|████████  | 1306/1621 [1:52:55<36:04,  6.87s/it]
 81%|████████  | 1307/1621 [1:53:00<32:32,  6.22s/it]
 81%|████████  | 1308/1621 [1:53:05<30:07,  5.77s/it]
 81%|████████  | 1309/1621 [1:53:09<28:31,  5.49s/it]
 81%|████████  | 1310/1621 [1:53:14<27:08,  5.24s/it]
                                                     

 81%|████████  | 1310/1621 [1:53:14<27:08,  5.24s/it]
 81%|████████  | 1311/1621 [1:53:19<26:20,  5.10s/it]
 81%|████████  | 1312/1621 [1:53:24<25:37,  4.98s/it]
 81%|████████  | 1313/1621 [1:53:30<27:05,  5.28s/it]
 81%|████████  | 1314/1621 [1:53:
0: {'loss': 0.2112, 'grad_norm': 0.27213095352265687, 'learning_rate': 1e-06, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.81}
0: 34<26:06,  5.10s/it]
 81%|████████  | 1315/1621 [1:53:39<25:27,  4.99s/it]
 81%|████████  | 1316/1621 [1:53:44<25:13,  4.96s/it]
 81%|████████  | 1317/1621 [1:53:49<25:07,  4.96s/it]
 81%|████████▏ | 1318/1621 [1:53:54<25:23,  5.03s/it]
 81%|████████▏ | 1319/1621 [1:53:59<24:46,  4.92s/it]
 81%|████████▏ | 1320/1621 [1:54:03<24:25,  4.87s/it]
                                                     

 81%|████████▏ | 1320/1621 [1:54:03<24:25,  4.87s/it]
 81%|████████▏ | 1321/1621 [1:54:08<24:02,  4.81s/it]
 82%|████████▏ | 1322/1621 [1:54:13<23:48,  4.78s/it]
 82%|████████▏ | 1323/1621 [1:54:18<23:48,  4.79s/it]
 82%|████████▏ | 1324/1621 [1:54:22<23:33,  4.76s/it]
 82%|████████▏ | 1325/1621 [1:54:27<23:40,  4.80s/it]
 82%|████████▏ | 1326/1621 [1:54:32<23:21,  4.75s/it]
 82%|████�
0: {'loss': 0.2114, 'grad_norm': 0.2722154666863481, 'learning_rate': 9.98421786662277e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.82}
0: ��███▏ | 1327/1621 [1:54:37<23:21,  4.77s/it]
 82%|████████▏ | 1328/1621 [1:54:42<23:31,  4.82s/it]
 82%|████████▏ | 1329/1621 [1:54:46<23:10,  4.76s/it]
 82%|████████▏ | 1330/1621 [1:54:52<24:50,  5.12s/it]
                                                     

 82%|████████▏ | 1330/1621 [1:54:52<24:50,  5.12s/it]
 82%|████████▏ | 1331/1621 [1:54:57<24:08,  5.00s/it]
 82%|████████▏ | 1332/1621 [1:55:02<23:42,  4.92s/it]
 82%|████████▏ | 1333/1621 [1:55:06<23:21,  4.87s/it]
 82%|████████▏ | 1334/1621 [1:55:11<23:03,  4.82s/it]
 82%|████████▏ | 1335/1621 [1:55:16<23:33,  4.94s/it]
 82%|████████▏ | 1336/1621 [1:55:21<23:06,  4.86s/it]
 82%|████████▏ | 1337/1621 [1:55:26<22:49,  4.82s/it]
 83%|████████▎ | 1338/1621 [1:55:31<22:52,  4.85s/it]
 83%|████████▎ | 1339/1621 [1:55:
0: {'loss': 0.2082, 'grad_norm': 0.2585298118477787, 'learning_rate': 9.9202926282791e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.83}
0: {'loss': 0.2067, 'grad_norm': 0.26664310644526223, 'learning_rate': 9.807937738894303e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.83}
0: 35<22:29,  4.79s/it]
 83%|████████▎ | 1340/1621 [1:55:40<22:39,  4.84s/it]
                                                     

 83%|████████▎ | 1340/1621 [1:55:40<22:39,  4.84s/it]
 83%|████████▎ | 1341/1621 [1:55:45<23:04,  4.94s/it]
 83%|████████▎ | 1342/1621 [1:55:51<24:02,  5.17s/it]
 83%|████████▎ | 1343/1621 [1:55:56<23:16,  5.02s/it]
 83%|████████▎ | 1344/1621 [1:56:01<22:43,  4.92s/it]
 83%|████████▎ | 1345/1621 [1:56:05<22:19,  4.85s/it]
 83%|████████▎ | 1346/1621 [1:56:10<22:10,  4.84s/it]
 83%|████████▎ | 1347/1621 [1:56:15<21:55,  4.80s/it]
 83%|████████▎ | 1348/1621 [1:56:19<21:41,  4.77s/it]
 83%|████████▎ | 1349/1621 [1:56:24<21:29,  4.74s/it]
 83%|████████▎ | 1350/1621 [1:56:29<21:40,  4.80s/it]
                                                     

 83%|████████
0: {'loss': 0.2053, 'grad_norm': 0.28712495588574566, 'learning_rate': 9.648384182148252e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.84}
0: ▎ | 1350/1621 [1:56:29<21:40,  4.80s/it]
 83%|████████▎ | 1351/1621 [1:56:34<21:41,  4.82s/it]
 83%|████████▎ | 1352/1621 [1:56:39<21:52,  4.88s/it]
 83%|████████▎ | 1353/1621 [1:56:44<21:33,  4.83s/it]
 84%|████████▎ | 1354/1621 [1:56:49<22:00,  4.95s/it]
 84%|████████▎ | 1355/1621 [1:56:54<22:50,  5.15s/it]
 84%|████████▎ | 1356/1621 [1:56:59<22:11,  5.02s/it]
 84%|████████▎ | 1357/1621 [1:57:04<22:12,  5.05s/it]
 84%|████████▍ | 1358/1621 [1:57:09<22:04,  5.04s/it]
 84%|████████▍ | 1359/1621 [1:57:14<21:32,  4.93s/it]
 84%|████████▍ | 1360/1621 [1:57:19<21:11,  4.87s/it]
                                                     

 84%|████████▍ | 1360/1621 [1:57:19<21:11,  4.87s/it]
 84%|████████▍ | 1361/1621 [1:57:24<21:02,  4.86s/it]
 84%|████████▍ | 1362/1621 [1:57:29<21:26,  
0: {'loss': 0.2077, 'grad_norm': 0.26673734209176836, 'learning_rate': 9.443380060197385e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.85}
0: 4.97s/it]
 84%|████████▍ | 1363/1621 [1:57:33<21:01,  4.89s/it]
 84%|████████▍ | 1364/1621 [1:57:39<21:58,  5.13s/it]
 84%|████████▍ | 1365/1621 [1:57:44<21:52,  5.13s/it]
 84%|████████▍ | 1366/1621 [1:57:49<21:44,  5.12s/it]
 84%|████████▍ | 1367/1621 [1:57:55<21:56,  5.18s/it]
 84%|████████▍ | 1368/1621 [1:58:00<21:36,  5.13s/it]
 84%|████████▍ | 1369/1621 [1:58:04<21:01,  5.00s/it]
 85%|████████▍ | 1370/1621 [1:58:09<20:56,  5.00s/it]
                                                     

 85%|████████▍ | 1370/1621 [1:58:09<20:56,  5.00s/it]
 85%|████████▍ | 1371/1621 [1:58:14<20:38,  4.95s/it]
 85%|████████▍ | 1372/1621 [1:58:19<20:18,  4.89s/it]
 85%|████████▍ | 1373/1621 [1:58:24<20:09,  4.88s/it]
 85%|████████▍ | 1374/1621 [1:58:29<19:50,  4.82s/it]
 85%|██████
0: {'loss': 0.214, 'grad_norm': 0.2619901325108232, 'learning_rate': 9.195171441101668e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.85}
0: ██▍ | 1375/1621 [1:58:34<20:14,  4.94s/it]
 85%|████████▍ | 1376/1621 [1:58:39<20:04,  4.92s/it]
 85%|████████▍ | 1377/1621 [1:58:43<19:46,  4.86s/it]
 85%|████████▌ | 1378/1621 [1:58:48<19:39,  4.85s/it]
 85%|████████▌ | 1379/1621 [1:58:53<19:30,  4.84s/it]
 85%|████████▌ | 1380/1621 [1:58:58<19:12,  4.78s/it]
                                                     

 85%|████████▌ | 1380/1621 [1:58:58<19:12,  4.78s/it]
 85%|████████▌ | 1381/1621 [1:59:02<19:02,  4.76s/it]
 85%|████████▌ | 1382/1621 [1:59:07<18:56,  4.76s/it]
 85%|████████▌ | 1383/1621 [1:59:12<18:55,  4.77s/it]
 85%|████████▌ | 1384/1621 [1:59:17<18:50,  4.77s/it]
 85%|████████▌ | 1385/1621 [1:59:22<19:12,  4.88s/it]
 86%|████████▌ | 1386/1621 [1:59:27<18:54,  4.83s/it]
 86%|████████▌ | 1387/1621 [1:59:31<18
0: {'loss': 0.207, 'grad_norm': 0.2769211257765989, 'learning_rate': 8.906477750432903e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.86}
0: :44,  4.80s/it]
 86%|████████▌ | 1388/1621 [1:59:36<18:34,  4.78s/it]
 86%|████████▌ | 1389/1621 [1:59:41<18:19,  4.74s/it]
 86%|████████▌ | 1390/1621 [1:59:45<18:15,  4.74s/it]
                                                     

 86%|████████▌ | 1390/1621 [1:59:45<18:15,  4.74s/it]
 86%|████████▌ | 1391/1621 [1:59:50<18:06,  4.73s/it]
 86%|████████▌ | 1392/1621 [1:59:55<17:59,  4.72s/it]
 86%|████████▌ | 1393/1621 [1:59:59<17:51,  4.70s/it]
 86%|████████▌ | 1394/1621 [2:00:04<17:51,  4.72s/it]
 86%|████████▌ | 1395/1621 [2:00:09<17:51,  4.74s/it]
 86%|████████▌ | 1396/1621 [2:00:14<17:45,  4.74s/it]
 86%|████████▌ | 1397/1621 [2:00:18<17:37,  4.72s/it]
 86%|████████▌ | 1398/1621 [2:00:23<17:31,  4.72s/it]
 86%|████████▋ | 1399/1621 [2:00:28<17:33,  4.74s/it]
 86%|████
0: {'loss': 0.2157, 'grad_norm': 0.2825136647523626, 'learning_rate': 8.580461976679099e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.86}
0: {'loss': 0.2117, 'grad_norm': 0.25441419447548025, 'learning_rate': 8.220696016880687e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.87}
0: ████▋ | 1400/1621 [2:00:33<17:26,  4.73s/it]
                                                     

 86%|████████▋ | 1400/1621 [2:00:33<17:26,  4.73s/it]
 86%|████████▋ | 1401/1621 [2:00:37<17:22,  4.74s/it]
 86%|████████▋ | 1402/1621 [2:00:42<17:14,  4.72s/it]
 87%|████████▋ | 1403/1621 [2:00:47<17:20,  4.78s/it]
 87%|████████▋ | 1404/1621 [2:00:52<17:09,  4.74s/it]
 87%|████████▋ | 1405/1621 [2:00:56<17:08,  4.76s/it]
 87%|████████▋ | 1406/1621 [2:01:01<16:58,  4.74s/it]
 87%|████████▋ | 1407/1621 [2:01:06<17:10,  4.82s/it]
 87%|████████▋ | 1408/1621 [2:01:11<17:00,  4.79s/it]
 87%|████████▋ | 1409/1621 [2:01:16<16:59,  4.81s/it]
 87%|████████▋ | 1410/1621 [2:01:20<16:51,  4.79s/it]
                                                     

 87%|████████▋ | 1410/1621 [2:01:20<16:51,  4.79s
0: {'loss': 0.2081, 'grad_norm': 0.27992518282615286, 'learning_rate': 7.831121542179086e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.88}
0: /it]
 87%|████████▋ | 1411/1621 [2:01:25<16:53,  4.83s/it]
 87%|████████▋ | 1412/1621 [2:01:31<17:23,  4.99s/it]
 87%|████████▋ | 1413/1621 [2:01:35<16:56,  4.89s/it]
 87%|████████▋ | 1414/1621 [2:01:41<17:29,  5.07s/it]
 87%|████████▋ | 1415/1621 [2:01:47<18:03,  5.26s/it]
 87%|████████▋ | 1416/1621 [2:01:51<17:26,  5.10s/it]
 87%|████████▋ | 1417/1621 [2:01:56<17:16,  5.08s/it]
 87%|████████▋ | 1418/1621 [2:02:01<17:04,  5.05s/it]
 88%|████████▊ | 1419/1621 [2:02:06<16:39,  4.95s/it]
 88%|████████▊ | 1420/1621 [2:02:11<16:20,  4.88s/it]
                                                     

 88%|████████▊ | 1420/1621 [2:02:11<16:20,  4.88s/it]
 88%|████████▊ | 1421/1621 [2:02:15<16:05,  4.83s/it]
 88%|████████▊ | 1422/1621 [2:02:20<16:00,  4.82s/it]
 88%|███████�
0: {'loss': 0.2102, 'grad_norm': 0.2623614381561981, 'learning_rate': 7.416006812042827e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.88}
0: �▊ | 1423/1621 [2:02:25<15:58,  4.84s/it]
 88%|████████▊ | 1424/1621 [2:02:30<15:42,  4.78s/it]
 88%|████████▊ | 1425/1621 [2:02:35<15:35,  4.77s/it]
 88%|████████▊ | 1426/1621 [2:02:40<15:43,  4.84s/it]
 88%|████████▊ | 1427/1621 [2:02:45<16:42,  5.17s/it]
 88%|████████▊ | 1428/1621 [2:02:50<16:25,  5.11s/it]
 88%|████████▊ | 1429/1621 [2:02:56<16:22,  5.12s/it]
 88%|████████▊ | 1430/1621 [2:03:00<15:56,  5.01s/it]
                                                     

 88%|████████▊ | 1430/1621 [2:03:00<15:56,  5.01s/it]
 88%|████████▊ | 1431/1621 [2:03:05<15:29,  4.89s/it]
 88%|████████▊ | 1432/1621 [2:03:10<15:34,  4.95s/it]
 88%|████████▊ | 1433/1621 [2:03:15<15:36,  4.98s/it]
 88%|████████▊ | 1434/1621 [2:03:20<15:21,  4.93s/it]
 89%|████████▊ | 1435/1621 [2:03:25<15:03, 
0: {'loss': 0.2116, 'grad_norm': 0.2687063278495978, 'learning_rate': 6.979899910323624e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.89}
0:  4.86s/it]
 89%|████████▊ | 1436/1621 [2:03:29<14:54,  4.84s/it]
 89%|████████▊ | 1437/1621 [2:03:34<15:01,  4.90s/it]
 89%|████████▊ | 1438/1621 [2:03:39<14:44,  4.83s/it]
 89%|████████▉ | 1439/1621 [2:03:44<14:35,  4.81s/it]
 89%|████████▉ | 1440/1621 [2:03:49<14:26,  4.79s/it]
                                                     

 89%|████████▉ | 1440/1621 [2:03:49<14:26,  4.79s/it]
 89%|████████▉ | 1441/1621 [2:03:53<14:18,  4.77s/it]
 89%|████████▉ | 1442/1621 [2:03:58<14:11,  4.76s/it]
 89%|████████▉ | 1443/1621 [2:04:03<14:08,  4.77s/it]
 89%|████████▉ | 1444/1621 [2:04:08<14:03,  4.77s/it]
 89%|████████▉ | 1445/1621 [2:04:12<13:57,  4.76s/it]
 89%|████████▉ | 1446/1621 [2:04:17<13:55,  4.77s/it]
 89%|████████▉ | 1447/1621 [2:04:22<13:51,  4.78s/it]
 89%|█████�
0: {'loss': 0.2102, 'grad_norm': 0.2714307919796291, 'learning_rate': 6.527578915497951e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.89}
0: �██▉ | 1448/1621 [2:04:27<13:44,  4.77s/it]
 89%|████████▉ | 1449/1621 [2:04:31<13:34,  4.74s/it]
 89%|████████▉ | 1450/1621 [2:04:36<13:30,  4.74s/it]
                                                     

 89%|████████▉ | 1450/1621 [2:04:36<13:30,  4.74s/it]
 90%|████████▉ | 1451/1621 [2:04:41<13:23,  4.73s/it]
 90%|████████▉ | 1452/1621 [2:04:46<13:34,  4.82s/it]
 90%|████████▉ | 1453/1621 [2:04:51<13:24,  4.79s/it]
 90%|████████▉ | 1454/1621 [2:04:56<13:27,  4.84s/it]
 90%|████████▉ | 1455/1621 [2:05:01<13:44,  4.97s/it]
 90%|████████▉ | 1456/1621 [2:05:05<13:25,  4.88s/it]
 90%|████████▉ | 1457/1621 [2:05:10<13:09,  4.82s/it]
 90%|████████▉ | 1458/1621 [2:05:15<13:01,  4.80s/it]
 90%|█████████ | 1459/1621 [2:05:20<13:18,  4.93s/it]
 90%|█████████ | 1460/1621 [2:05:25<1
0: {'loss': 0.2083, 'grad_norm': 0.2627481844715906, 'learning_rate': 6.06399955103937e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.9}
0: {'loss': 0.2042, 'grad_norm': 0.2713225358577235, 'learning_rate': 5.594240889475106e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.91}
0: 3:04,  4.87s/it]
                                                     

 90%|█████████ | 1460/1621 [2:05:25<13:04,  4.87s/it]
 90%|█████████ | 1461/1621 [2:05:30<12:50,  4.82s/it]
 90%|█████████ | 1462/1621 [2:05:35<13:01,  4.91s/it]
 90%|█████████ | 1463/1621 [2:05:40<12:56,  4.91s/it]
 90%|█████████ | 1464/1621 [2:05:44<12:39,  4.84s/it]
 90%|█████████ | 1465/1621 [2:05:49<12:25,  4.78s/it]
 90%|█████████ | 1466/1621 [2:05:54<12:24,  4.80s/it]
 90%|█████████ | 1467/1621 [2:05:58<12:15,  4.78s/it]
 91%|█████████ | 1468/1621 [2:06:03<12:06,  4.75s/it]
 91%|█████████ | 1469/1621 [2:06:08<11:57,  4.72s/it]
 91%|█████████ | 1470/1621 [2:06:13<11:52,  4.72s/it]
                                                     

 91%|█████████ | 1470/1621 [2:06:13<11:52,  4.72s/it]
 91%|█████████ 
0: {'loss': 0.2089, 'grad_norm': 0.2537827651441038, 'learning_rate': 5.123449705004581e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.91}
0: | 1471/1621 [2:06:17<11:51,  4.74s/it]
 91%|█████████ | 1472/1621 [2:06:22<12:04,  4.86s/it]
 91%|█████████ | 1473/1621 [2:06:27<11:52,  4.82s/it]
 91%|█████████ | 1474/1621 [2:06:33<12:18,  5.02s/it]
 91%|█████████ | 1475/1621 [2:06:37<11:59,  4.93s/it]
 91%|█████████ | 1476/1621 [2:06:43<12:07,  5.02s/it]
 91%|█████████ | 1477/1621 [2:06:47<11:47,  4.92s/it]
 91%|█████████ | 1478/1621 [2:06:52<11:32,  4.85s/it]
 91%|█████████ | 1479/1621 [2:06:57<11:49,  5.00s/it]
 91%|█████████▏| 1480/1621 [2:07:02<11:43,  4.99s/it]
                                                     

 91%|█████████▏| 1480/1621 [2:07:02<11:43,  4.99s/it]
 91%|█████████▏| 1481/1621 [2:07:07<11:23,  4.89s/it]
 91%|█████████▏| 1482/1621 [2:07:12<11:18,  4.88s/it]
 91%|█████████▏| 1483/1621 [2:07:17<11
0: {'loss': 0.2087, 'grad_norm': 0.2823914222814313, 'learning_rate': 4.656784084364238e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.92}
0: :07,  4.83s/it]
 92%|█████████▏| 1484/1621 [2:07:22<11:14,  4.92s/it]
 92%|█████████▏| 1485/1621 [2:07:26<11:01,  4.86s/it]
 92%|█████████▏| 1486/1621 [2:07:31<10:55,  4.86s/it]
 92%|█████████▏| 1487/1621 [2:07:36<10:47,  4.83s/it]
 92%|█████████▏| 1488/1621 [2:07:41<10:54,  4.92s/it]
 92%|█████████▏| 1489/1621 [2:07:46<10:39,  4.84s/it]
 92%|█████████▏| 1490/1621 [2:07:51<10:29,  4.81s/it]
                                                     

 92%|█████████▏| 1490/1621 [2:07:51<10:29,  4.81s/it]
 92%|█████████▏| 1491/1621 [2:07:55<10:21,  4.78s/it]
 92%|█████████▏| 1492/1621 [2:08:00<10:13,  4.75s/it]
 92%|█████████▏| 1493/1621 [2:08:05<10:08,  4.75s/it]
 92%|█████████▏| 1494/1621 [2:08:10<10:11,  4.82s/it]
 92%|█████████▏| 1495/1621 [2:08:15<10:12,  4
0: {'loss': 0.21, 'grad_norm': 0.2831683932276976, 'learning_rate': 4.1993569137498776e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.93}
0: .86s/it]
 92%|█████████▏| 1496/1621 [2:08:20<10:36,  5.09s/it]
 92%|█████████▏| 1497/1621 [2:08:25<10:17,  4.98s/it]
 92%|█████████▏| 1498/1621 [2:08:30<10:10,  4.96s/it]
 92%|█████████▏| 1499/1621 [2:08:35<09:55,  4.88s/it]
 93%|█████████▎| 1500/1621 [2:08:39<09:43,  4.82s/it]
                                                     

 93%|█████████▎| 1500/1621 [2:08:39<09:43,  4.82s/it]
 93%|█████████▎| 1501/1621 [2:08:44<09:32,  4.77s/it]
 93%|█████████▎| 1502/1621 [2:08:49<09:24,  4.74s/it]
 93%|█████████▎| 1503/1621 [2:08:54<09:28,  4.82s/it]
 93%|█████████▎| 1504/1621 [2:08:58<09:18,  4.77s/it]
 93%|█████████▎| 1505/1621 [2:09:03<09:21,  4.84s/it]
 93%|█████████▎| 1506/1621 [2:09:08<09:13,  4.81s/it]
 93%|█████████▎| 1507/1621 [2:09:13<09:04,  4.77s/it
0: {'loss': 0.2151, 'grad_norm': 0.2947685205241263, 'learning_rate': 3.7561798609655373e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.93}
0: ]
 93%|█████████▎| 1508/1621 [2:09:17<08:59,  4.77s/it]
 93%|█████████▎| 1509/1621 [2:09:23<09:08,  4.90s/it]
 93%|█████████▎| 1510/1621 [2:09:28<09:06,  4.92s/it]
                                                     

 93%|█████████▎| 1510/1621 [2:09:28<09:06,  4.92s/it]
 93%|█████████▎| 1511/1621 [2:09:32<08:54,  4.86s/it]
 93%|█████████▎| 1512/1621 [2:09:38<09:01,  4.97s/it]
 93%|█████████▎| 1513/1621 [2:09:42<08:47,  4.88s/it]
 93%|█████████▎| 1514/1621 [2:09:48<09:06,  5.11s/it]
 93%|█████████▎| 1515/1621 [2:09:53<08:52,  5.03s/it]
 94%|█████████▎| 1516/1621 [2:09:58<08:49,  5.04s/it]
 94%|█████████▎| 1517/1621 [2:10:03<09:02,  5.22s/it]
 94%|█████████▎| 1518/1621 [2:10:08<08:49,  5.14s/it]
 94%|█████████▎| 1519/1621 [2:10:13<08:32,  5.02s/it]
 94%|
0: {'loss': 0.2045, 'grad_norm': 0.2678544345260796, 'learning_rate': 3.3321084665422803e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.94}
0: {'loss': 0.2093, 'grad_norm': 0.27176220768733284, 'learning_rate': 2.931788945420058e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.94}
0: █████████▍| 1520/1621 [2:10:18<08:35,  5.10s/it]
                                                     

 94%|█████████▍| 1520/1621 [2:10:18<08:35,  5.10s/it]
 94%|█████████▍| 1521/1621 [2:10:23<08:24,  5.04s/it]
 94%|█████████▍| 1522/1621 [2:10:28<08:19,  5.04s/it]
 94%|█████████▍| 1523/1621 [2:10:34<08:42,  5.34s/it]
 94%|█████████▍| 1524/1621 [2:10:39<08:17,  5.13s/it]
 94%|█████████▍| 1525/1621 [2:10:44<07:57,  4.98s/it]
 94%|█████████▍| 1526/1621 [2:10:48<07:43,  4.87s/it]
 94%|█████████▍| 1527/1621 [2:10:53<07:33,  4.82s/it]
 94%|█████████▍| 1528/1621 [2:10:58<07:29,  4.83s/it]
 94%|█████████▍| 1529/1621 [2:11:03<07:20,  4.79s/it]
 94%|█████████▍| 1530/1621 [2:11:08<07:25,  4.89s/it]
                                                     

 94%|████████�
0: {'loss': 0.2121, 'grad_norm': 0.2805922111643405, 'learning_rate': 2.5596072820445254e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.95}
0: �▍| 1530/1621 [2:11:08<07:25,  4.89s/it]
 94%|█████████▍| 1531/1621 [2:11:13<07:29,  4.99s/it]
 95%|█████████▍| 1532/1621 [2:11:18<07:21,  4.96s/it]
 95%|█████████▍| 1533/1621 [2:11:22<07:10,  4.89s/it]
 95%|█████████▍| 1534/1621 [2:11:27<07:03,  4.87s/it]
 95%|█████████▍| 1535/1621 [2:11:32<06:55,  4.83s/it]
 95%|█████████▍| 1536/1621 [2:11:37<06:47,  4.79s/it]
 95%|█████████▍| 1537/1621 [2:11:41<06:39,  4.75s/it]
 95%|█████████▍| 1538/1621 [2:11:46<06:36,  4.78s/it]
 95%|█████████▍| 1539/1621 [2:11:51<06:30,  4.77s/it]
 95%|█████████▌| 1540/1621 [2:11:56<06:35,  4.88s/it]
                                                     

 95%|█████████▌| 1540/1621 [2:11:56<06:35,  4.88s/it]
 95%|█████████▌| 1541/1621 [2:12:01<06:29,  4.87s/it]
 95%|█████████▌| 1
0: {'loss': 0.2115, 'grad_norm': 0.4089994863899589, 'learning_rate': 2.2196411766036487e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.96}
0: 542/1621 [2:12:07<06:44,  5.13s/it]
 95%|█████████▌| 1543/1621 [2:12:11<06:31,  5.02s/it]
 95%|█████████▌| 1544/1621 [2:12:16<06:19,  4.92s/it]
 95%|█████████▌| 1545/1621 [2:12:21<06:09,  4.87s/it]
 95%|█████████▌| 1546/1621 [2:12:26<06:02,  4.83s/it]
 95%|█████████▌| 1547/1621 [2:12:30<05:53,  4.78s/it]
 95%|█████████▌| 1548/1621 [2:12:35<05:47,  4.75s/it]
 96%|█████████▌| 1549/1621 [2:12:40<05:48,  4.84s/it]
 96%|█████████▌| 1550/1621 [2:12:46<05:57,  5.03s/it]
                                                     

 96%|█████████▌| 1550/1621 [2:12:46<05:57,  5.03s/it]
 96%|█████████▌| 1551/1621 [2:12:50<05:45,  4.93s/it]
 96%|█████████▌| 1552/1621 [2:12:55<05:35,  4.86s/it]
 96%|█████████▌| 1553/1621 [2:13:01<05:55,  5.23s/it]
 96%|█████████▌| 1554/162
0: {'loss': 0.2144, 'grad_norm': 0.2651691403897037, 'learning_rate': 1.915615368891117e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.96}
0: 1 [2:13:06<05:38,  5.05s/it]
 96%|█████████▌| 1555/1621 [2:13:11<05:33,  5.05s/it]
 96%|█████████▌| 1556/1621 [2:13:15<05:20,  4.94s/it]
 96%|█████████▌| 1557/1621 [2:13:20<05:13,  4.89s/it]
 96%|█████████▌| 1558/1621 [2:13:25<05:08,  4.89s/it]
 96%|█████████▌| 1559/1621 [2:13:31<05:21,  5.19s/it]
 96%|█████████▌| 1560/1621 [2:13:36<05:06,  5.03s/it]
                                                     

 96%|█████████▌| 1560/1621 [2:13:36<05:06,  5.03s/it]
 96%|█████████▋| 1561/1621 [2:13:41<05:10,  5.18s/it]
 96%|█████████▋| 1562/1621 [2:13:46<04:58,  5.05s/it]
 96%|█████████▋| 1563/1621 [2:13:51<04:46,  4.94s/it]
 96%|█████████▋| 1564/1621 [2:13:55<04:41,  4.93s/it]
 97%|█████████▋| 1565/1621 [2:14:00<04:31,  4.86s/it]
 97%|█████████▋| 1566/1621 [2:14
0: {'loss': 0.2132, 'grad_norm': 0.2650836750987023, 'learning_rate': 1.6508608292777203e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.97}
0: :05<04:25,  4.82s/it]
 97%|█████████▋| 1567/1621 [2:14:10<04:17,  4.76s/it]
 97%|█████████▋| 1568/1621 [2:14:14<04:11,  4.74s/it]
 97%|█████████▋| 1569/1621 [2:14:19<04:06,  4.75s/it]
 97%|█████████▋| 1570/1621 [2:14:24<04:05,  4.82s/it]
                                                     

 97%|█████████▋| 1570/1621 [2:14:24<04:05,  4.82s/it]
 97%|█████████▋| 1571/1621 [2:14:29<03:59,  4.78s/it]
 97%|█████████▋| 1572/1621 [2:14:33<03:53,  4.76s/it]
 97%|█████████▋| 1573/1621 [2:14:38<03:48,  4.77s/it]
 97%|█████████▋| 1574/1621 [2:14:44<03:52,  4.95s/it]
 97%|█████████▋| 1575/1621 [2:14:48<03:44,  4.87s/it]
 97%|█████████▋| 1576/1621 [2:14:53<03:36,  4.81s/it]
 97%|█████████▋| 1577/1621 [2:14:58<03:30,  4.78s/it]
 97%|█████████▋| 1578/1621 [2:15:02<03:
0: {'loss': 0.2101, 'grad_norm': 0.2573210674484348, 'learning_rate': 1.4282782639029128e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.97}
0: 25,  4.78s/it]
 97%|█████████▋| 1579/1621 [2:15:07<03:18,  4.74s/it]
 97%|█████████▋| 1580/1621 [2:15:12<03:12,  4.71s/it]
                                                     

 97%|█████████▋| 1580/1621 [2:15:12<03:12,  4.71s/it]
 98%|█████████▊| 1581/1621 [2:15:17<03:11,  4.79s/it]
 98%|█████████▊| 1582/1621 [2:15:21<03:04,  4.74s/it]
 98%|█████████▊| 1583/1621 [2:15:26<03:00,  4.75s/it]
 98%|█████████▊| 1584/1621 [2:15:31<03:00,  4.88s/it]
 98%|█████████▊| 1585/1621 [2:15:37<03:06,  5.18s/it]
 98%|█████████▊| 1586/1621 [2:15:42<02:59,  5.13s/it]
 98%|█████████▊| 1587/1621 [2:15:47<02:51,  5.03s/it]
 98%|█████████▊| 1588/1621 [2:15:52<02:42,  4.93s/it]
 98%|█████████▊| 1589/1621 [2:15:56<02:35,  4.86s/it]
 98%|█████████▊| 1590/1621 [2:16:01<02:30,  4.
0: {'loss': 0.2086, 'grad_norm': 0.2553344270166335, 'learning_rate': 1.2503063339313356e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.98}
0: {'loss': 0.206, 'grad_norm': 0.25174882817793, 'learning_rate': 1.1188949370707787e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.99}
0: 85s/it]
                                                     

 98%|█████████▊| 1590/1621 [2:16:01<02:30,  4.85s/it]
 98%|█████████▊| 1591/1621 [2:16:06<02:24,  4.81s/it]
 98%|█████████▊| 1592/1621 [2:16:11<02:21,  4.89s/it]
 98%|█████████▊| 1593/1621 [2:16:16<02:19,  4.97s/it]
 98%|█████████▊| 1594/1621 [2:16:21<02:13,  4.93s/it]
 98%|█████████▊| 1595/1621 [2:16:26<02:10,  5.01s/it]
 98%|█████████▊| 1596/1621 [2:16:31<02:07,  5.11s/it]
 99%|█████████▊| 1597/1621 [2:16:36<02:00,  5.01s/it]
 99%|█████████▊| 1598/1621 [2:16:41<01:55,  5.04s/it]
 99%|█████████▊| 1599/1621 [2:16:46<01:48,  4.94s/it]
 99%|█████████▊| 1600/1621 [2:16:52<01:48,  5.16s/it]
                                                     

 99%|█████████▊| 1600/1621 [2:16:52<01:48,  5.16s/it]
 99%|████�
0: {'loss': 0.2112, 'grad_norm': 0.2496232881123691, 'learning_rate': 1.0354838440848501e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 0.99}
0: ��████▉| 1601/1621 [2:16:56<01:40,  5.01s/it]
 99%|█████████▉| 1602/1621 [2:17:01<01:33,  4.93s/it]
 99%|█████████▉| 1603/1621 [2:17:06<01:27,  4.87s/it]
 99%|█████████▉| 1604/1621 [2:17:11<01:23,  4.88s/it]
 99%|█████████▉| 1605/1621 [2:17:16<01:18,  4.88s/it]
 99%|█████████▉| 1606/1621 [2:17:20<01:12,  4.83s/it]
 99%|█████████▉| 1607/1621 [2:17:25<01:07,  4.85s/it]
 99%|█████████▉| 1608/1621 [2:17:30<01:04,  4.96s/it]
 99%|█████████▉| 1609/1621 [2:17:35<00:58,  4.88s/it]
 99%|█████████▉| 1610/1621 [2:17:40<00:53,  4.83s/it]
                                                     

 99%|█████████▉| 1610/1621 [2:17:40<00:53,  4.83s/it]
 99%|█████████▉| 1611/1621 [2:17:45<00:47,  4.77s/it]
 99%|█████████▉| 1612/1621 [2:17:49<00:42,  4.76s/it]
100%|██████�
0: {'loss': 0.2094, 'grad_norm': 0.2608941458825701, 'learning_rate': 1.0009869243631952e-07, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 1.0}
0: [2025-09-02 21:23:01,404] [INFO] [axolotl.core.trainers.base._save:613] [PID:2134977] [RANK:0] Saving model checkpoint to /lustre/fswork/projects/rech/dgo/udv55np/math/Qwen3-235B-A22B/Qwen2.5-14B_ift/0/checkpoint-1621
0: [2025-09-02 21:23:23,920] [INFO] [axolotl.core.trainers.base._save:662] [PID:2134977] [RANK:0] Saving Trainer.data_collator.tokenizer by default as Trainer.processing_class is `None`
0: {'train_runtime': 8360.3247, 'train_samples_per_second': 3.102, 'train_steps_per_second': 0.194, 'train_loss': 0.21736425671549509, 'memory/max_mem_active(gib)': 53.9, 'memory/max_mem_allocated(gib)': 51.85, 'memory/device_mem_reserved(gib)': 57.41, 'epoch': 1.0}
0: �██▉| 1613/1621 [2:17:54<00:38,  4.76s/it]
100%|█████████▉| 1614/1621 [2:17:59<00:33,  4.73s/it]
100%|█████████▉| 1615/1621 [2:18:03<00:28,  4.73s/it]
100%|█████████▉| 1616/1621 [2:18:08<00:23,  4.73s/it]
100%|█████████▉| 1617/1621 [2:18:13<00:18,  4.70s/it]
100%|█████████▉| 1618/1621 [2:18:18<00:14,  4.98s/it]
100%|█████████▉| 1619/1621 [2:18:23<00:09,  4.90s/it]
100%|█████████▉| 1620/1621 [2:18:28<00:04,  4.85s/it]
                                                     

100%|█████████▉| 1620/1621 [2:18:28<00:04,  4.85s/it]
100%|██████████| 1621/1621 [2:18:39<00:00,  6.66s/it]
                                                     

100%|██████████| 1621/1621 [2:19:20<00:00,  6.66s/it]
100%|██████████| 1621/1621 [2:19:20<00:00,  5.16s/it]
0: [2025-09-02 21:23:35,467] [INFO] [axolotl.train.save_trained_model:228] [PID:2134977] [RANK:0] Training completed! Saving trained model to /lustre/fswork/projects/rech/dgo/udv55np/math/Qwen3-235B-A22B/Qwen2.5-14B_ift/0.
0: [2025-09-02 21:23:42,461] [INFO] [axolotl.core.trainers.base._save:613] [PID:2134977] [RANK:0] Saving model checkpoint to /lustre/fswork/projects/rech/dgo/udv55np/math/Qwen3-235B-A22B/Qwen2.5-14B_ift/0
0: [2025-09-02 21:24:04,610] [INFO] [axolotl.core.trainers.base._save:662] [PID:2134977] [RANK:0] Saving Trainer.data_collator.tokenizer by default as Trainer.processing_class is `None`
0: [2025-09-02 21:24:05,016] [INFO] [axolotl.train.save_trained_model:350] [PID:2134977] [RANK:0] Model successfully saved to /lustre/fswork/projects/rech/dgo/udv55np/math/Qwen3-235B-A22B/Qwen2.5-14B_ift/0