- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents WIMW (Was It Me or the World?), a framework for scope-aware adaptation in embodied agents. The central problem is how an agent should determine the appropriate scope of a restriction after observing an execution failure, rather than repeatedly encountering the same limitation across subsequent tasks.The paper considers three possible explanations for failures: skill-wide capability limitations (Body), target-type-specific restrictions (World), and transient disruptions. Since an isolated failure does not necessarily distinguish these cases, directly generalizing such observations into persistent memory may lead to either overrestriction or underrestriction.
To address this problem, WIMW collects diagnostic evidence through target retries, cross-type reference tests, and final target verification. The resulting restrictions are encoded as capability contracts and enforced by an execution guard across tasks. The authors further introduce Patch-VOI, a decision-theoretic acquisition criterion that selects diagnostic actions based on their expected reduction in scope-decision loss while accounting for interaction and verification costs.The framework is evaluated with a frozen Qwen3-VL-32B planner on EB-ALFRED and EB-Habitat under injected pick failures. The experiments include comparisons with existing memory methods, executable-memory baselines, passive scope inference, and alternative diagnostic acquisition strategies.
Overall, the paper aims to move beyond passive failure memory toward active scope inference and enforceable cross-task behavioral restrictions.
Critical strengths
1: Meaningful problem formulation and a coherent motivation for executable failure memory
The strongest aspect of the paper is its formulation of failure memory as a scope-selection problem.Existing approaches to agent memory often emphasize how to summarize, store, and retrieve experiences. This paper asks a more specific question: how broadly should an observed failure affect future actions? This perspective highlights an important trade-off. Overly broad restrictions may suppress feasible actions, whereas overly narrow restrictions may allow the same underlying limitation to cause repeated failures across tasks.
The proposed capability contracts and execution guard provide a concrete mechanism for translating diagnostic conclusions into persistent behavioral constraints, rather than relying solely on a language model to follow textual advice.
I find this formulation practically meaningful, and the overall progression from scope uncertainty to diagnosis, verification, and execution-time enforcement provides a coherent research motivation.
2: Reasonably systematic evaluation with efforts to isolate the contributions of different components
The experimental study goes beyond a straightforward comparison against No update or general-purpose memory baselines.In particular, the authors introduce persistent executable caches and a matched Passive global + R baseline to investigate the additional value of reference acquisition while controlling for recovery, verification, execution guarding, and termination behavior.
Another positive aspect is that the paper distinguishes unexplained failures, task completion, total interactions, and scope-decision loss. The authors generally acknowledge that reducing failed attempts does not necessarily imply improved task completion.
Critical weaknesses
1.The main diagnostic mechanisms rely on relatively direct heuristic rules, while Patch-VOI is primarily built upon existing decision theory. More importantly, the connection between the theoretical methods and the main complete system experiments is not close enough, affecting the paper's technical innovation and overall consistency.
2.Current experiments mainly demonstrate that explicit behavioral constraints can reduce repeated failures under predefined fault conditions, but lack sufficient analysis of the correctness of scope inference, diagnostic decision-making mechanisms, and complex fault conditions. This means that the paper's core research claims have not been sufficiently validated.
Questions for authors
- Relationship between the reference controller and numerical Patch-VOI (clarification).
Could the authors clarify why the main full-stream experiments evaluate the rule-based reference controller, while numerical Patch-VOI is assessed separately under a different, shorter task-stream protocol? What is the intended methodological relationship between these two controllers, and what specific new insight does Patch-VOI provide beyond the reference diagnostic rules? In particular, would the advantages of numerical Patch-VOI remain when evaluated under the same full-stream execution protocol?
- Direct validation of scope-inference quality (clarification/additional analysis).
Could the authors provide a direct evaluation of the accuracy of inferred capability contracts, including false restrictions on feasible actions and failures to restrict genuinely unavailable actions? It would also be useful to analyze how different reference-test outcomes affect the final contract decision. More fundamentally, how does the proposed type-level restriction handle failures that differ across instances of the same object type? Under what assumptions is final verification on the triggering target sufficient to justify a type-wide restriction?
- Robustness and interpretability of active diagnostic gains (additional analysis).
Could the authors provide uncertainty estimates for the main paired comparisons in Table 3(b) and Table 4, together with a more fine-grained breakdown across fault conditions? How sensitive is numerical Patch-VOI to the relative penalties for overrestriction and underrestriction, as well as the diagnostic-cost weight? Under what conditions does Patch-VOI make substantially different or better decisions than fixed-order acquisition or simpler heuristic strategies?
4 comments
Part 2 of 2
Q2. Correctness of the inferred contracts.
The contracts are correct. In every stream behind Table 3(a), WIMW stored exactly the contract the injected fault requires and blocked no feasible action. When instances of a type fail differently, the error is a small over-restriction.
We scored every stored contract and blocked action against the injected fault in the 40 streams behind Table 3(a). Each Body stream stored one skill-wide contract, each World stream one type-local contract on the faulty type, and Transient and Clean streams none. With the rates of Appendix A.2, WIMW leaves 3.1% / 2.6% of disabled demands uncovered, all before its first diagnosis, against 21.4% / 18.7% for the persistent cache. In the 12-task blocks of Table 4, where scope is ambiguous by design, Patch-VOI has the lowest rate of both errors. It blocks 1.2% / 1.0% of feasible demands, against 3.4% / 2.9% for the fixed order and 2.7% / 3.1% for information gain, and leaves 12.8% / 12.1% uncovered, against 14.2% / 13.7% and 15.3% / 13.9%.
The reference outcomes select the scope, and the target re-check decides whether anything is stored. References that also fail point to a skill-wide fault, and a reference success points to a type-local one. Under Transient faults, 80 diagnoses saw a reference succeed, and the re-check rejected all 80. Failures with a stated cause, such as "not near the robot", never count as evidence.
Verifying only the trigger target suffices when failures recur at the level at which actions are named. EmbodiedBench actions name object types, so the guard blocks a type, and the re-check, a fresh find and pick of that type, tests recurrence at that level. When we injected the fault on half or a quarter of a type's instances, type contracts formed in 6 of 11 and 3 of 13 diagnoses, against 4 of 4 when all instances fail, because the re-check passes whenever it reaches a working instance. A stored contract then blocks 3.5 / 2.5 and 1.5 / 0.5 feasible picks per stream, and WIMW still has fewer unexplained failures than no memory (7.25 against 15.5, 5.5 against 8.0). An interface that names instances would change only .
Q3. Uncertainty and sensitivity.
The main comparisons hold under paired 95% intervals, and Patch-VOI's advantage holds at every cost weight we tried.
In Table 3(a), under Body, WIMW has 27.5 ± 0.7 unexplained failures against 72.5 ± 6.4 for the persistent cache in ALFRED, and 25.0 ± 0.0 against 58.5 ± 4.9 in Habitat. For Table 3(b) and Table 4, the paired differences in EB-ALFRED are below, with 95% bootstrap intervals over 16 paired streams. Negative values favor the first method.
| Paired difference | Body | World | Transient | Clean |
|---|---|---|---|---|
| , matched WIMW − Passive global + R | −3.9 [−5.1, −2.8] | 0.0 [0.0, 0.0] | −0.1 [−0.6, 0.4] | 0.0 |
| , Patch-VOI − fixed order | −0.33 [−0.51, −0.16] | −0.23 [−0.38, −0.09] | −0.12 [−0.21, −0.04] | 0.00 |
| , Patch-VOI − information gain | −0.55 [−0.78, −0.33] | −0.44 [−0.63, −0.26] | −0.37 [−0.52, −0.23] | 0.00 |
| , Patch-VOI − verification only | −6.88 [−8.17, −5.64] | −2.51 [−3.22, −1.83] | −0.97 [−1.41, −0.56] | 0.00 |
The intervals exclude zero wherever the paper reports a gain, and reference acquisition helps only under Body, as Section 5 states. EB-Habitat has the same signs (condition means −0.13, −0.29 and −2.22), and Clean triggers no diagnosis.
We varied one weight at a time around the paper values (, ), and Patch-VOI stays ahead of the fixed order at every setting, with all intervals excluding zero. Its advantage grows with the over-restriction weight, from −0.06 at a ratio of 1 to −0.29 at 10, while its false-restriction rate falls from 2.7% to 0.6%. It also grows with the action cost, from −0.09 at to −0.34 at 0.3, because the fixed order keeps running tests that cannot change the contract. Patch-VOI therefore departs from the fixed order most when wrong contracts or tests are expensive: in 26.3% of diagnoses at the paper weights, 33.8% at a ratio of 10 and 38.1% at . For the reference controller, varying each of its four thresholds left every contract in Table 3(a) unchanged. Failures moved by at most 1.5 per stream, except under the recheck frequency, which trades failures under a persistent fault against slower recovery once the fault clears.
We thank the reviewer again and would be grateful if these results lead to a reconsideration of the score.
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to clarify the relationship between the reference controller and Patch-VOI and to strengthen the empirical evaluation. However, my main concerns regarding the generalizability of the approach remain. The predefined fault taxonomy closely matches WIMW's hypothesis space, making it unclear whether the method can generalize to more complex or unseen failure structures. The reliance on simulator metadata further limits its applicability to realistic embodied settings.
While I acknowledge the improved empirical evidence and diagnostic efficiency demonstrated in the rebuttal, these results primarily establish effectiveness within structured fault settings rather than generalizable failure-scope adaptation. Therefore, my concerns about the scope of the contribution and its applicability beyond the predefined fault assumptions remain, and I maintain my original rating.
We thank the reviewer for the follow-up and for recognizing the clarified relationship between the two controllers and the stronger evidence. The remaining concern, whether WIMW generalizes beyond faults that match its hypotheses, is a fair question for any method that reasons over a structured hypothesis space, and we would like to add two points that we hope are helpful.
The hypothesis space follows from what a memory can store, not from the faults we injected. A stored restriction has to block some set of future actions, and with an action interface that names skills and object types, that set is the whole skill, some object types, or nothing. Whatever the cause of a failure, whether joint wear, a slippery material or an occluding shelf, the memory has to choose among these sets. B, W_c and N are therefore the restrictions the agent can express, and the injected faults test whether it chooses among them correctly.
Several of the new conditions lie outside the three hypotheses, and in each WIMW either adapts correctly or errs on the safe side:
- A limitation on a whole object class covers several types but not the whole skill, which none of B, W_c or N describes. WIMW restricted the affected types one at a time and never extended the restriction to the skill, reducing unexplained failures from 132.0 / 138.0 to 14.0 / 22.0.
- A persistent fault overlaid with transient failures in every episode mixes two hypotheses. WIMW restricted only the persistently failing type, with 59.5 / 59.0 failures against 94.0 / 109.5 without memory and 54.0 / 54.0 for the oracle.
- A fault on only some instances of a type cannot be expressed by any type-level contract. The re-check stores a contract less often as the faulty share drops, the over-restriction stays at most 3.5 / 2.5 feasible picks per stream, and WIMW still has fewer failures than no memory.
- Faults that change during a 300-task stream were all covered by contracts of the correct scope.
- The Clean streams contain 422–529 / 1,337–1,509 natural failures that we did not design. Even with the trigger filter removed, none of them produced a contract.
We also tested the diagnosis without simulator metadata. The agent read metadata for two signals, the objects present and the hand state. We replaced them with reference objects proposed by the same VLM from the camera image and with hand state tracked from action feedback. In EB-ALFRED the stored contracts were identical to those of the metadata version, and unexplained failures were unchanged (28.0 and 28.0 under Body, 3.0 and 3.0 under World). In EB-Habitat the agent can navigate only to receptacles, so without metadata it rarely reaches a second object to test. In that case WIMW stores no contract rather than a wrong one, which delays the restriction without causing over-restriction.
We agree that physical deployment and failure structures richer than skills and types are the natural next steps, and we will add a section on failures outside the hypothesis space, with the results above, to the revision. We hope these points clarify that WIMW does not depend on the fault matching its hypotheses, and that when it does not match, the error stays bounded. We would be grateful if the reviewer would take them into account in the final assessment, and we thank the reviewer again for the careful engagement throughout the discussion.
Part 1 of 2
We thank the reviewer for the careful reading and the specific questions. Our answers in brief are as follows. The reference controller and numerical Patch-VOI run the same diagnosis and differ only in how the next test is chosen, and Patch-VOI keeps its advantage when run in full streams. The contracts WIMW stores match the injected fault in every single-fault stream, and when only part of a type fails, the resulting error is a small over-restriction. The paired comparisons exclude zero wherever the paper reports a gain, and Patch-VOI's advantage holds at every cost weight we tried. The evidence follows, with results reported as EB-ALFRED / EB-Habitat.
Q1. Relationship between the two controllers, and Patch-VOI in full streams.
The two controllers are one procedure with two ways of choosing the next test. Under the full-stream protocol, Patch-VOI reaches the same contracts as the reference controller with fewer tests.
Both controllers retry the target, test reference objects, re-check the target before committing, and enforce the contract with the guard and periodic rechecks. The reference controller chooses the next test with two fixed rules taken from the analysis. A retry of the same target cannot separate a skill-wide fault from a type-local one (Prop. 1), so it tests other objects, and once a retry succeeds no further test can change the decision (Prop. 2), so it stops. Numerical Patch-VOI instead computes the choice from the expected reduction in scope loss minus the action cost (Eq. 6–7).
Patch-VOI adds a stopping rule tied to the stored decision. A test can reduce uncertainty about the fault and still leave the contract unchanged. Information gain keeps paying for such tests, whereas Patch-VOI skips them.
The paper evaluates Patch-VOI in 12-task blocks because a full stream gives the selector few decisions to make. WIMW commits a contract at the first unexplained failure, after which the guard blocks the failing action, so a stream with one persistent fault contains about one diagnosis. The blocks repeat this diagnosis with memory reset, which yields enough decisions to compare selectors.
In full streams with a single fault, Patch-VOI stores the same contracts with fewer tests. We placed it inside the deployed agent and reran the 60-task streams of Table 3(a), so both controllers share the planner, trigger, guard, recheck and step budget. Neither stores a wrong restriction. Patch-VOI needs 9.0 / 9.0 diagnostic actions against 11.0 / 10.5 under Body and 7.5 / 7.0 against 9.5 / 8.5 under World, and has 1.0 / 0.5 fewer unexplained failures under Body. Under Transient faults the two behave identically.
When a full stream contains repeated diagnoses, Patch-VOI still has the lowest cost. We built 300-task streams in which the fault changes every 60 tasks (Body, World on one type, Transient, World on a second type, Clean), ten per environment (two task orders × five executor seeds, Appendix B.4). Only test selection differs between arms, and J follows Eq. 13 with the paper weights, summed over the stream.
All three selectors store a contract of the correct scope for all 60 persistent segments and block the same 5.6 / 4.9 feasible picks per stream. Those blocks come from contracts that remain in force after their segment ends, until a recheck lifts them. The selectors therefore differ only in the cost of diagnosis. Patch-VOI uses 17.0 / 11.0 fewer diagnostic actions than the fixed order and leaves fewer demands uncovered, lowering J by 7.6% / 5.5%. Information gain spends the most actions and has the highest J, the same ordering as in Table 4. The reduction is smaller than the 9.1% / 7.7% in the blocks because the shared over-restriction term makes up about 70% of J here. We will add both comparisons to Section 5.