ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning
Paper • 2604.19254 • Published • 27
WhereIsAI
community
AI & ML interests
Where is AI🤔? It is here👆. We aim at building advanced Natural Language Processing techniques and employing them for applications.
Recent Activity
View all activity
SeanLee97
submitted a
paper to Daily Papers 2 days ago
csroyli
authored 4
papers 2 days ago
Label Supervised LLaMA Finetuning
Paper • 2310.01208 • Published
Recurrent Attention Networks for Long-text Modeling
Paper • 2306.06843 • Published
2D Matryoshka Sentence Embeddings
Paper • 2402.14776 • Published • 8
CondAmbigQA: A Benchmark and Dataset for Conditional Ambiguous Question Answering
Paper • 2502.01523 • Published • 2
SeanLee97
authored a
paper 2 days ago
csroyli
authored a
paper 2 days ago
Post
7857
Our lab recently released a paper where we introduce ShadowPEFT, a new Parameter-Efficient Fine-Tuning (PEFT) paradigm tailored for edge computing scenarios.
Unlike traditional approaches such as LoRA and its variants, which inject trainable parameters directly into the weights of Transformer, requiring tight coupling with the backbone.
ShadowPEFT instead enhances the frozen large base model by adding a lightweight, centralized, pretrainable, and detachable Shadow network.
This shadow network operates in parallel with the base model, delivering learned corrections to each decoder layer. Because the shadow module is architecturally decoupled from the backbone, it can be independently trained, stored, and deployed, benefiting edge computing scenarios and edge-cloud collaboration computing.
- HF Paper: ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning (2604.19254)
- GitHub: https://github.com/ShadowLLM/shadow-peft
- HF Collection: https://huggingface.co/collections/shadow-llm/shadow-peft-models
Unlike traditional approaches such as LoRA and its variants, which inject trainable parameters directly into the weights of Transformer, requiring tight coupling with the backbone.
ShadowPEFT instead enhances the frozen large base model by adding a lightweight, centralized, pretrainable, and detachable Shadow network.
This shadow network operates in parallel with the base model, delivering learned corrections to each decoder layer. Because the shadow module is architecturally decoupled from the backbone, it can be independently trained, stored, and deployed, benefiting edge computing scenarios and edge-cloud collaboration computing.
- HF Paper: ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning (2604.19254)
- GitHub: https://github.com/ShadowLLM/shadow-peft
- HF Collection: https://huggingface.co/collections/shadow-llm/shadow-peft-models
- 7 replies
SeanLee97
updated a
collection 7 months ago
add AIBOM
❤️ 1
#29 opened 9 months ago
by 
RiccardoDav
SeanLee97
published a
model 11 months ago
SeanLee97
updated a
Space 11 months ago
SeanLee97
updated a
model 11 months ago
SeanLee97
published a
dataset 11 months ago
SeanLee97
updated a
dataset 11 months ago
SeanLee97
updated a
model 11 months ago
SeanLee97
published a
Space 11 months ago
Validating fine-tuning of AnglE model
5
#27 opened about 1 year ago
by 
hugguc
Multiple crashes when training model WhereIsAI/UAE-Large-V1
7
#26 opened about 1 year ago
by hugguc
Encoding a large Knowledge Base
2
#25 opened over 1 year ago
by 
SachinVashistha