Was It Me or the World? Scope-Aware Adaptation for Embodied Agents

An embodied agent that relearns the same limitation on every task spends its interaction budget on avoidable failures. Yet an isolated failure does not reveal whether a capability is unavailable everywhere, restricted locally, or temporarily disrupted. We introduce WIMW ( Was It Me or the World?), a training-free method that uses diagnostic actions to determine the scope of future restrictions. Reference tests and target verification produce capability contracts that an execution guard enforces across tasks. Patch-VOI selects tests by their expected reduction in scope errors after verification, charging for the actions needed to obtain evidence. With a frozen Qwen3-VL-32B planner on EB-ALFRED and EB-Habitat, the reference controller reduces unexplained failed picks by 95.0% and 96.0% relative to No update under injected skill-wide faults across 300-task streams. Under these faults, matched comparisons against passive global inference show additional failure reductions of 12.3% and 13.3% at unchanged mean completion; actions rise by 2.6% in ALFRED and fall by 13.8% in Habitat. Numerical Patch-VOI lowers mean scope loss plus diagnostic cost by 9.1% and 7.7% relative to fixed-order acquisition, with slightly lower mean completion. Together, these results connect reusable failure memory to the decision of whether further evidence is worth its interaction cost. Code and data: https://anonymous.4open.science/r/WIMW-Anonymous-Code-858F.