|
|
--- |
|
|
license: apache-2.0 |
|
|
tags: |
|
|
- cvpr25 |
|
|
- JarvisIR |
|
|
- weights |
|
|
description: | |
|
|
This repository contains the official weights for the CVPR 2025 paper "JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restoration". |
|
|
--- |
|
|
|
|
|
# JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restoration |
|
|
|
|
|
## Model Description |
|
|
|
|
|
JarvisIR is a novel system that leverages a Vision-Language Model (VLM) to intelligently restore images for autonomous driving perception in adverse weather. It acts as a central controller, dynamically coordinating multiple expert restoration models to tackle complex degradations such as rain, fog, low-light, and snow. |
|
|
|
|
|
## Key Features |
|
|
|
|
|
- **VLM Controller**: The first framework to employ a Vision-Language Model for orchestrating image restoration workflows. |
|
|
- **Multi-Expert Coordination**: Dynamically schedules specialized restoration models for tasks like denoising, super-resolution, and deraining. |
|
|
- **Adaptive Restoration**: Effectively handles a wide range of adverse weather conditions, including night/low-light, rain, fog, and snow. |
|
|
- **Advanced Training Strategy**: Utilizes a two-stage process of Supervised Fine-Tuning (SFT) followed by alignment with Mixed-Rank Reward-based Human Feedback (MRRHF). |
|
|
|
|
|
## Model Architecture |
|
|
|
|
|
The system comprises three core components: |
|
|
|
|
|
1. **VLM Controller**: A LLaVA-v1.5-7B model serves as the core for task planning and expert model selection. |
|
|
2. **Expert Models**: A suite of specialized networks, each tailored for a specific restoration task (e.g., deraining, defogging). |
|
|
3. **Reward Models**: A set of Image Quality Assessment (IQA) models that provide feedback for quality assessment and alignment during training. |
|
|
|
|
|
## Training Data |
|
|
|
|
|
JarvisIR was trained on a large-scale, comprehensive dataset: |
|
|
|
|
|
- **CleanBench-Synthetic**: A dataset of 150,000 synthetically degraded images with corresponding annotations. |
|
|
- **CleanBench-Real**: A collection of 80,000 real-world images captured in adverse weather, used for alignment training. |
|
|
- **Comprehensive Coverage**: The data covers four primary weather scenarios (night, rain, fog, snow) with various combinations of degradations. |
|
|
|
|
|
## Performance |
|
|
|
|
|
- Achieves a **50% average improvement** in perception metrics on the CleanBench-Real dataset compared to state-of-the-art all-in-one methods. |
|
|
- Demonstrates superior performance across all tested weather conditions. |
|
|
- Exhibits enhanced robustness and generalization capabilities in real-world driving scenarios. |
|
|
|
|
|
## Intended Use |
|
|
|
|
|
**Primary Use Cases:** |
|
|
- Enhancing perception systems in autonomous vehicles. |
|
|
- Building robust, multi-weather image restoration pipelines. |
|
|
- Advancing research into the applications of Vision-Language Models in image processing. |
|
|
|
|
|
## Model Checkpoints |
|
|
|
|
|
This repository provides the following model weights: |
|
|
- `pertained`: The complete model after both Supervised Fine-Tuning and MRRHF alignment stages. |
|
|
- `agent-tools/`: The weights for each individual expert restoration model. |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you find JarvisIR useful in your research, please cite our paper: |
|
|
|
|
|
```bibtex |
|
|
@inproceedings{lin2025jarvisir, |
|
|
title={Jarvisir: Elevating autonomous driving perception with intelligent image restoration}, |
|
|
author={Lin, Yunlong and Lin, Zixu and Chen, Haoyu and Pan, Panwang and Li, Chenxin and Chen, Sixiang and Wen, Kairun and Jin, Yeying and Li, Wenbo and Ding, Xinghao}, |
|
|
booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference}, |
|
|
pages={22369--22380}, |
|
|
year={2025} |
|
|
} |
|
|
``` |
|
|
|
|
|
## Related Resources |
|
|
|
|
|
- **Project Page**: https://cvpr2025-jarvisir.github.io/ |
|
|
- **Code Repository**: https://github.com/LYL1015/JarvisIR |
|
|
- **Paper**: https://arxiv.org/pdf/2504.04158 |
|
|
|
|
|
## Acknowledgments |
|
|
|
|
|
This work contributes to the advancement of intelligent image restoration by integrating Vision-Language Models with expert system coordination. |
