|
|
--- |
|
|
library_name: transformers |
|
|
tags: |
|
|
- llama-3 |
|
|
- code-generation |
|
|
- qlora |
|
|
- peft |
|
|
- colab |
|
|
license: llama3 |
|
|
datasets: |
|
|
- codeparrot/conala-mined-curated |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- meta-llama/Meta-Llama-3-8B |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# Model Card for llama3-codeweaver-lora |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Model name:** llama3-codeweaver-lora |
|
|
- **Developed by:** [mahmoudalrefaey](https://huggingface.co/mahmoudalrefaey) |
|
|
- **Funded by:** None (personal project) |
|
|
- **Finetuned from:** [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) |
|
|
- **License:** LLaMA 3 license |
|
|
|
|
|
This is a **LLaMA-3 8B model fine-tuned with QLoRA** on the [CoNaLa mined-curated dataset](https://huggingface.co/datasets/codeparrot/conala-mined-curated) for **code generation tasks**. |
|
|
The adapter was trained on **Google Colab T4 (16GB)** using **fp16 mixed precision** with QLoRA for efficiency. |
|
|
|
|
|
--- |
|
|
|
|
|
## Uses |
|
|
|
|
|
### Direct Use |
|
|
- Intended for **code generation assistant tasks** such as transforming natural language instructions into Python snippets. |
|
|
- Educational use for learning about LLM fine-tuning with LoRA adapters. |
|
|
|
|
|
### Downstream Use |
|
|
- Can be further fine-tuned on specialized coding datasets (e.g. SQL, JS). |
|
|
- Integration into coding assistants and research projects. |
|
|
|
|
|
### Out-of-Scope Use |
|
|
- Not intended for production-critical code security auditing. |
|
|
- Not guaranteed to generate safe or fully optimized code. |
|
|
- Should not be used in environments where code execution safety is critical without sandboxing. |
|
|
|
|
|
--- |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Training Data |
|
|
- Dataset: [CoNaLa mined-curated](https://huggingface.co/datasets/codeparrot/conala-mined-curated) |
|
|
- Dataset size used: ~7,000 samples |
|
|
|
|
|
### Training Procedure |
|
|
- **Method:** QLoRA fine-tuning with 4-bit quantization |
|
|
- **Precision:** fp16 mixed precision |
|
|
- **Hardware:** Google Colab T4 (16GB GPU) |
|
|
- **Batch size:** 2 → effective batch 4 with accumulation |
|
|
- **Epochs:** 3 |
|
|
- **Training time:** ~1h 30m |
|
|
|
|
|
--- |
|
|
|
|
|
## Evaluation |
|
|
|
|
|
### Testing Data |
|
|
- Held-out validation split (10% of dataset) |
|
|
|
|
|
### Metrics |
|
|
- **Validation Loss** decreased steadily across epochs |
|
|
- **Qualitative Evaluation:** Generated Python snippets from validation prompts |
|
|
- Example outputs matched reference solutions for common coding tasks |
|
|
|
|
|
### Example Prompt & Output |
|
|
``` |
|
|
Prompt: |
|
|
### Instruction: |
|
|
Write code to convert integer num to list |
|
|
|
|
|
### Code: |
|
|
|
|
|
Generated: |
|
|
[int(x) for x in str(num)] |
|
|
``` |
|
|
|
|
|
## Environmental Impact |
|
|
|
|
|
- Hardware: NVIDIA T4 (16 GB VRAM) |
|
|
- Cloud Provider: Google Colab |
|
|
- Compute Region: Unknown |
|
|
- Training Duration: ~1.5 hours |
|
|
|
|
|
## Citation |
|
|
@misc{llama3-codeweaver-lora, |
|
|
author = {Mahmoud Alrefaey}, |
|
|
title = {llama3-codeweaver-lora: A QLoRA fine-tuned LLaMA-3 model for code generation}, |
|
|
year = {2025}, |
|
|
publisher = {Hugging Face}, |
|
|
howpublished = {\url{https://huggingface.co/mahmoudalrefaey/llama3-codeweaver-lora}}, |
|
|
} |
