Continual GUI Agents
Abstract
Continual GUI Agents framework addresses performance degradation in dynamic digital environments through reinforcement fine-tuning with novel anchoring rewards that stabilize learning across shifting UI domains and resolutions.
As digital environments (data distribution) are in flux, with new GUI data arriving over time-introducing new domains or resolutions-agents trained on static environments deteriorate in performance. In this work, we introduce Continual GUI Agents, a new task that requires GUI agents to perform continual learning under shifted domains and resolutions. We find existing methods fail to maintain stable grounding as GUI distributions shift over time, due to the diversity of UI interaction points and regions in fluxing scenarios. To address this, we introduce GUI-Anchoring in Flux (GUI-AiF), a new reinforcement fine-tuning framework that stabilizes continual learning through two novel rewards: Anchoring Point Reward in Flux (APR-iF) and Anchoring Region Reward in Flux (ARR-iF). These rewards guide the agents to align with shifting interaction points and regions, mitigating the tendency of existing reward strategies to over-adapt to static grounding cues (e.g., fixed coordinates or element scales). Extensive experiments show GUI-AiF surpasses state-of-the-art baselines. Our work establishes the first continual learning framework for GUI agents, revealing the untapped potential of reinforcement fine-tuning for continual GUI Agents.
Community
Some of the observations founded are :-
-- Static GUI training breaks under real world change :
GUI agents trained on fixed datasets degrade badly when UI domains (mobile --> desktop --> web) or resolutions (1080p -> 4K) shift, mainly due to unstable grounding of interaction points and regions.
-- SFT overfits ; RFT is more suitable for continual learning :
Supervised Fine Tuning memorizes current layouts and forgets prior skills, while Reinforcement Fine Tuning ( RFT ) better preserves knowledge via KL regularized updates making it a stronger base for continual GUI agents.
-- Grounding fails because interaction points and scales are in flux :
Domain and resolution shifts cause large changes in element locations and sizes, and existing reward designs over adapt to static coordinates or scales, leading to poor generalization.
-- GUI AiF stabilizes learning by rewarding diversity, not fixation :
The proposed GUI AiF framework introduces two rewards APR-iF ( diverse interaction points ) and ARR-iF ( diverse element regions ) which prevent agents from collapsing onto single layouts.
-- Generalizing interaction points matters more than scale :
Ablations show APR-iF contributes more than ARR-iF, indicating that adapting to where to interact is more critical than adapting to how big elements are in continual GUI environments.
..... many more...
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Reflux (2026)
- MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolution (2026)
- MobileDreamer: Generative Sketch World Model for GUI Agent (2026)
- OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution (2026)
- GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Models (2026)
- Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning (2025)
- EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experience (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper