winglian/qwen3-4b-math

See axolotl config

axolotl version: 0.10.0.dev0

base_model: Qwen/Qwen3-4B-Base

plugins:
  - axolotl.integrations.liger.LigerPlugin
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin

liger_rms_norm: true
liger_glu_activation: true

# torch_compile: true

dataloader_prefetch_factor: 4
dataloader_num_workers: 2
dataloader_pin_memory: true

chat_template: qwen3
datasets:
  - path: winglian/OpenThoughts-114k-math-correct
    type: chat_template
    split: train
    split_thinking: true
    eot_tokens:
      - "<|im_end|>"

dataset_prepared_path: last_run_prepared
val_set_size: 0.01
output_dir: ./outputs/model-out-math-4b

sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true

wandb_project: kd-4b-math
wandb_entity: axolotl-ai
wandb_name: sft-4b

gradient_accumulation_steps: 1
micro_batch_size: 4
num_epochs: 2
optimizer: adamw_torch_fused
adam_beta2: 0.95
lr_scheduler: rex
learning_rate: 3e-5
max_grad_norm: 0.1
save_safetensors: true

bf16: true
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
logging_steps: 1
flash_attention: true

warmup_steps: 100
evals_per_epoch: 4
saves_per_epoch: 1
weight_decay: 0.1
special_tokens:
  eos_token: <|im_end|>
deepspeed: deepspeed_configs/zero2_torch_compile.json

outputs/model-out-math-4b

This model is a fine-tuned version of Qwen/Qwen3-4B-Base on the winglian/OpenThoughts-114k-math-correct dataset. It achieves the following results on the evaluation set:

• Loss: 0.3929

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 32
• total_eval_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.95) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 100
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss
0.5644	0.0016	1	0.5801
0.4038	0.2504	159	0.4154
0.3914	0.5008	318	0.4035
0.3812	0.7512	477	0.3960
0.3626	1.0016	636	0.3915
0.316	1.2520	795	0.3958
0.3171	1.5024	954	0.3963
0.2944	1.7528	1113	0.3929

Framework versions

• Transformers 4.51.3
• Pytorch 2.7.0+cu128
• Datasets 3.5.1
• Tokenizers 0.21.1