README.md

---
base_model: meta-llama/Meta-Llama-3-8B
library_name: peft
license: llama3
datasets:
- lewtun/github-issues
language:
- en
pipeline_tag: summarization
---

# Model Card: LoRA-LLaMA3-8B-GitHub-Summarizer

This repository provides LoRA adapter weights fine-tuned on top of Meta’s LLaMA-3-8B model for the task of summarizing GitHub issues and discussions. The model was trained on a curated dataset of open-source GitHub issues to produce concise, readable, and technically accurate summaries.

## Model Details

### Model Description

- **Developed by:** Saramsh Gautam (Louisiana State University)
- **Model type:** LoRA adapter weights
- **Language(s):** English
- **License:** llama (must comply with Meta's license)
- **Fine-tuned from model:** `meta-llama/Meta-Llama-3-8B`
- **Library used:** PEFT (LoRA) with Hugging Face Transformers

### Model Sources

- **Base model:** [Meta-LLaMA-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B)
- **Repository:** [link to this repo]()

## Uses

### Direct Use

These adapter weights must be merged with the base LLaMA-3-8B model using PEFT or Hugging Face’s `PeftModel` wrapper.

Example use case:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel, PeftConfig

base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B")
model = PeftModel.from_pretrained(base_model, "saramshgautam/lora-llama-8b-github")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
```

### Intended USe

- Research in summarization of technical conversations

- Augmenting code review and issue tracking pipelines

- Studying model adaptation via parameter-efficient fine-tuning

### Out-of-Scope Use

- Commercial applications (restricted by Meta’s LLaMA license)

- General-purpose conversation or chatbot use (model optimized for summarization)

## Bias, Risks, and Limitations

- The model inherits biases from both the base LLaMA-3 model and the GitHub dataset. It may underperform on non-technical content or multilingual issues.

## Recommendations

Use only for academic or non-commercial research. Evaluate responsibly before using in production or public-facing tools.

## How to Get Started with the Model

See the example in “Direct Use” above. You must separately download the base model from Meta and load the LoRA adapters from this repo.

## Training Details

### Training Data

- Source: Hugging Face lewtun/github-issues
- Description: Contains 3,000+ GitHub issues and comments from popular open-source repositories.

## Training Procedure

- LoRA with PEFT
- 4-bit quantized training using bitsandbytes
- Mixed precision: bf16
- Batch size: 8
- Epochs: 3
- Optimizer: AdamW

### Evaluation

## Metrics

ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-Lsum on a 500-issue test set

## Results

| Metric     | Score |
| ---------- | ----- |
| ROUGE-1    | 0.706 |
| ROUGE-2    | 0.490 |
| ROUGE-L    | 0.570 |
| ROUGE-Lsum | 0.582 |

---

## Environmental Impact

- **Hardware Type:** 4×A100 GPUs (university HPC cluster)
- **Training Hours:** ~4 hours
- **Carbon Estimate:** ~10.2 kg CO₂eq  
  _(estimated via [ML CO2 calculator](https://mlco2.github.io/impact))_

---

## Citation

**APA:**

Gautam, S. (2025). _LoRA-LLaMA3-8B-GitHub-Summarizer: Adapter weights for summarizing GitHub issues using LLaMA 3_. Hugging Face. https://huggingface.co/saramshgautam/lora-llama-8b-github

**BibTeX:**

```bibtex
@misc{gautam2025lora,
  title={LoRA-LLaMA3-8B-GitHub-Summarizer},
  author={Gautam, Saramsh},
  year={2025},
  howpublished={\url{https://huggingface.co/saramshgautam/lora-llama-8b-github}},
  note={Fine-tuned adapter weights using LoRA on Meta-LLaMA-3-8B}
}
```

---

## Contact

- **Author:** Saramsh Gautam
- **Affiliation:** Louisiana State University
- **Email:** [your email]
- **Hugging Face profile:** [https://huggingface.co/saramshgautam](https://huggingface.co/saramshgautam)

---

## Framework Versions

- **PEFT:** 0.15.2
- **Transformers:** 4.40.0
- **Bitsandbytes:** 0.41.3
- **Datasets:** 2.18.0