README.md

---
library_name: transformers
tags:
- llama-3
- code-generation
- qlora
- peft
- colab
license: llama3
datasets:
- codeparrot/conala-mined-curated
language:
- en
base_model:
- meta-llama/Meta-Llama-3-8B
pipeline_tag: text-generation
---

# Model Card for llama3-codeweaver-lora

## Model Details

- **Model name:** llama3-codeweaver-lora  
- **Developed by:** [mahmoudalrefaey](https://huggingface.co/mahmoudalrefaey)  
- **Funded by:** None (personal project)  
- **Finetuned from:** [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B)  
- **License:** LLaMA 3 license  

This is a **LLaMA-3 8B model fine-tuned with QLoRA** on the [CoNaLa mined-curated dataset](https://huggingface.co/datasets/codeparrot/conala-mined-curated) for **code generation tasks**.  
The adapter was trained on **Google Colab T4 (16GB)** using **fp16 mixed precision** with QLoRA for efficiency.  

---

## Uses

### Direct Use
- Intended for **code generation assistant tasks** such as transforming natural language instructions into Python snippets.  
- Educational use for learning about LLM fine-tuning with LoRA adapters.  

### Downstream Use
- Can be further fine-tuned on specialized coding datasets (e.g. SQL, JS).  
- Integration into coding assistants and research projects.  

### Out-of-Scope Use
- Not intended for production-critical code security auditing.  
- Not guaranteed to generate safe or fully optimized code.  
- Should not be used in environments where code execution safety is critical without sandboxing.  

---

## Training Details

### Training Data
- Dataset: [CoNaLa mined-curated](https://huggingface.co/datasets/codeparrot/conala-mined-curated)  
- Dataset size used: ~7,000 samples  

### Training Procedure
- **Method:** QLoRA fine-tuning with 4-bit quantization  
- **Precision:** fp16 mixed precision  
- **Hardware:** Google Colab T4 (16GB GPU)  
- **Batch size:** 2 → effective batch 4 with accumulation  
- **Epochs:** 3  
- **Training time:** ~1h 30m  

---

## Evaluation

### Testing Data
- Held-out validation split (10% of dataset)  

### Metrics
- **Validation Loss** decreased steadily across epochs  
- **Qualitative Evaluation:** Generated Python snippets from validation prompts  
- Example outputs matched reference solutions for common coding tasks  

### Example Prompt & Output
```
Prompt:
### Instruction:
Write code to convert integer num to list

### Code:

Generated:
[int(x) for x in str(num)]
```

## Environmental Impact

- Hardware: NVIDIA T4 (16 GB VRAM)
- Cloud Provider: Google Colab
- Compute Region: Unknown
- Training Duration: ~1.5 hours

## Citation
@misc{llama3-codeweaver-lora,
  author = {Mahmoud Alrefaey},
  title = {llama3-codeweaver-lora: A QLoRA fine-tuned LLaMA-3 model for code generation},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/mahmoudalrefaey/llama3-codeweaver-lora}},
}