---
license: apache-2.0
tags:
- smollm
- python
- code-generation
- instruct
- qlora
- fine-tuned
- code
- nf4
datasets:
- flytech/python-codes-25k
model-index:
- name: HF-SmolLM-1.7B-0.5B-4bit-coder
  results: []
language:
- en
pipeline_tag: text-generation
---

# HF-SmolLM-1.7B-0.5B-4bit-coder

## Model Summary
**HF-SmolLM-1.7B-0.5B-4bit-coder** is a fine-tuned variant of [SmolLM-1.7B](https://huggingface.co/HuggingFaceTB/SmolLM-1.7B), optimized for **instruction-following in Python code generation tasks**.  
It was trained on a **1,500-sample subset** of the [flytech/python-codes-25k](https://huggingface.co/datasets/flytech/python-codes-25k) dataset using **parameter-efficient fine-tuning (QLoRA 4-bit)**.

The model is suitable for:
- Generating Python code snippets from natural language instructions  
- Completing short code functions  
- Educational prototyping of fine-tuned LMs  

⚠️ This is **not a production-ready coding assistant**. Generated outputs must be manually reviewed before execution.  

---

## Intended Uses & Limitations

### ✅ Intended
- Research on parameter-efficient fine-tuning  
- Educational demos of instruction-tuning workflows  
- Prototype code generation experiments  

### ❌ Not Intended
- Deployment in production coding assistants  
- Safety-critical applications  
- Long-context multi-file programming tasks  

---

## Training Details

### Base Model
- **Name:** [HuggingFaceTB/SmolLM-1.7B](https://huggingface.co/HuggingFaceTB/SmolLM-1.7B)  
- **Architecture:** Decoder-only causal LM  
- **Total Parameters:** 1.72B  
- **Fine-tuned Trainable Parameters:** ~9M (0.53%)  

### Dataset
- **Source:** [flytech/python-codes-25k](https://huggingface.co/datasets/flytech/python-codes-25k)  
- **Subset Used:** 1,500 randomly sampled examples  
- **Content:** Instruction + optional input → Python code output  
- **Formatting:** Converted into `chat` format with `user` / `assistant` roles  

### Training Procedure
- **Framework:** Hugging Face Transformers + TRL (SFTTrainer)  
- **Quantization:** 4-bit QLoRA (nf4) with bfloat16 compute when available  
- **Effective Batch Size:** 6 (with accumulation)  
- **Optimizer:** AdamW  
- **Scheduler:** Cosine decay with warmup ratio 0.05  
- **Epochs:** 3  
- **Learning Rate:** 2e-4  
- **Max Seq Length:** 64 tokens (training)  
- **Mixed Precision:** FP16  
- **Gradient Checkpointing:** Enabled  

---

## Evaluation
No formal benchmark evaluation has been conducted yet.  
Empirically, the model:
- Produces syntactically valid Python code for simple tasks  
- Adheres to given instructions with reasonable accuracy  
- Struggles with multi-step reasoning and long code outputs  

---

## Example Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "sweatSmile/HF-SmolLM-1.7B-0.5B-4bit-coder"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, device_map="auto")

prompt = "Write a Python function that checks if a number is prime."
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    return_tensors="pt",
    add_generation_prompt=True
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=150)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))