duyddinh351/Lisa-LoRA-Llama3.1-8B-v1

Lisa-Visa-LoRA-Llama-v2

A LoRA fine-tuned variant of Llama 3.1–8B Instruct, specialized for Vietnamese travel visa consultation.
This model powers “Lisa”, the AI assistant for SaigonTimes Travel, designed to provide natural, accurate, and structured guidance for visa applications.

---

📌 1. Model Summary

• Base model: meta-llama/Llama-3.1-8B-Instruct
• Finetuning method: LoRA
• Language: Vietnamese (primary), English (secondary)
• Domain: Tourist visa, financial proof, employment proof, interview preparation
• Style: Friendly, helpful, professional consultant tone
• Intended use:
  • Visa Q&A chatbot
  • Automated support for travel agencies
  • Multi-turn visa advisory
  • Handling weak cases (low income, new savings, limited travel history)

---

📌 2. Model Description

Lisa-Visa-LoRA-Llama-v2 is a LoRA adapter trained on a curated visa-focused dataset containing 4,600 conversational samples, including:

• Single-turn Q&A
• Multi-turn dialogues
• Case-based reasoning
• Document checklists
• Policy summaries
• Interview situation simulation

The model improves:

• Coherence in Vietnamese
• Accuracy for visa requirements
• Tone consistency for customer support
• Ability to reason for weak profiles
• Reduction in hallucination vs. base model

---

📌 3. Intended Use Cases

✔ Recommended

• Visa consultation chatbots
• AI assistant for travel companies
• Automated customer support
• Checklist generation & document explanation
• Interview preparation simulation

❌ Not recommended

• Immigration, study abroad, or work visa (not trained)
• Legal or guaranteed visa approval prediction
• Handling sensitive documents without human review

---

📌 4. Training Dataset

The model was fine-tuned on 4,600 hand-curated and synthetic visa-related samples, formatted in ChatML (messages format).

Dataset sources:

• Internal SaigonTimes Travel visa conversation logs
• Public Vietnamese Q&A rewritten for quality
• Reddit data related to visa & travel
• Synthetic expansions using self-instruct prompts

Dataset split:

Split	Count
Train	80%
Validation	10%
Test	10%

Public dataset:
👉 duyddinh351/visa-dataset

---

📌 5. Training Procedure

Hyperparameters

Parameter	Value
LoRA Rank	64
LoRA Alpha	16
LoRA Dropout	0.05
Bits	4-bit NF4
LR	2e-4
Scheduler	Cosine
Epochs	3
Max seq length	4,096

Hardware

• 1 × NVIDIA A100 80GB
• Training time: ~45 minutes

Libraries

• Transformers
• PEFT
• BitsAndBytes
• vLLM (inference)

---

📌 6. Evaluation

In-domain visa QA (internal test set)

• Accuracy: ~89%
• Coherence (human eval): 8.4/10
• Tone consistency: 9/10
• Hallucination reduction: ~25% vs. base model

Note: This model is domain-specialized; generic LLM benchmarks (MMLU, GSM8K) are not relevant.

---

📌 7. Limitations

• Visa policies change frequently → responses may become outdated
• Not suitable for legal/immigration consultation
• May provide overconfident reasoning if prompt is unclear
• Should not be used to guarantee visa approval

---

📌 8. Ethical Considerations

• Should not promote document fraud
• Should not replace human visa experts for critical cases
• Users must verify final requirements with official embassy sources

---

📌 9. How to Use (Transformers + PEFT)

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "meta-llama/Llama-3.1-8B-Instruct"
lora = "duyddinh351/Lisa-Visa-LoRA-Llama-v2"

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, lora)

prompt = "What documents do I need to apply for a Japan tourist visa?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=300)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

base_model: meta-llama/Llama-3.1-8B-Instruct
library_name: peft
model_name: LISA_llma3.1_v2
tags:
- base_model:adapter:meta-llama/Llama-3.1-8B-Instruct
- lora
- sft
- transformers
- trl
licence: license
pipeline_tag: text-generation
---

# Model Card for LISA_llma3.1_v2

This model is a fine-tuned version of [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct).
It has been trained using [TRL](https://github.com/huggingface/trl).

## Quick start

```python
from transformers import pipeline

question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
generator = pipeline("text-generation", model="None", device="cuda")
output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
print(output["generated_text"])

Training procedure

This model was trained with SFT.

Framework versions

• PEFT 0.17.1
• TRL: 0.23.0
• Transformers: 4.56.2
• Pytorch: 2.8.0
• Datasets: 4.3.0
• Tokenizers: 0.22.1

Citations

Cite TRL as:

@misc{vonwerra2022trl,
    title        = {{TRL: Transformer Reinforcement Learning}},
    author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
    year         = 2020,
    journal      = {GitHub repository},
    publisher    = {GitHub},
    howpublished = {\url{https://github.com/huggingface/trl}}
}