CAPAgent: Keeping Spatial Agents Reliable When Their Perception Models Change

Spatial embodied agents rely on external perception models for depth, pose, and grounding to make task decisions, and these models are often replaced after the agent is trained. Similar readings from different models can differ in how far they can be trusted, so the right action can change after a replacement. This paper asks how one trained agent can make correct decisions with unseen perception models under fixed task rules and interfaces, without updating its weights. The agent judges reliability from quality cues, such as how widely depth values spread around a doorway, but a cue rises both at real scene edges and when the model is noisier. We show that an agent trained on one output at a time, from many scenes but few models, learns mostly the scene-driven meaning of these cues and underreacts when a new model raises them. We first build CapSwap-Bench, a benchmark that replays actual model replacements on matched scenes, with switch pairs where the right action changes and retain pairs where it should not. We then present CAPAgent, which trains on outputs of two perception models for the same scene and query, so that their difference isolates the model change. The core idea is to learn how its decision margins should move within such pairs, crossing a task threshold only when the right action changes; at deployment, CAPAgent still reads one output at a time. When three tool models are swapped for unseen ones, CAPAgent-8B reaches an average score of 70.6 on five standard spatial benchmarks, retains 96% of its original-tool score, and matches the 27B SpatialCLI agent. On CapSwap-Bench, it acts correctly on 84.7% of switch pairs and 92.3% of retain pairs. Anonymous code repository: https://anonymous.4open.science/r/CAPAgent-8E4C/.