acceptodds
Under review as a conference paper at ICLR 2027

Was It Me or the World? Scope-Aware Adaptation for Embodied Agents

Abstract

An embodied agent that relearns the same limitation on every task spends its interaction budget on avoidable failures. Yet an isolated failure does not reveal whether a capability is unavailable everywhere, restricted locally, or temporarily disrupted. We introduce WIMW ( Was It Me or the World?), a training-free method that uses diagnostic actions to determine the scope of future restrictions. Reference tests and target verification produce capability contracts that an execution guard enforces across tasks. Patch-VOI selects tests by their expected reduction in scope errors after verification, charging for the actions needed to obtain evidence. With a frozen Qwen3-VL-32B planner on EB-ALFRED and EB-Habitat, the reference controller reduces unexplained failed picks by 95.0% and 96.0% relative to No update under injected skill-wide faults across 300-task streams. Under these faults, matched comparisons against passive global inference show additional failure reductions of 12.3% and 13.3% at unchanged mean completion; actions rise by 2.6% in ALFRED and fall by 13.8% in Habitat. Numerical Patch-VOI lowers mean scope loss plus diagnostic cost by 9.1% and 7.7% relative to fixed-order acquisition, with slightly lower mean completion. Together, these results connect reusable failure memory to the decision of whether further evidence is worth its interaction cost. Code and data: https://anonymous.4open.science/r/WIMW-Anonymous-Code-858F.

Primary area:
applications to robotics, autonomy, planning
Keywords:
embodied agents, online adaptation, agent memory, active diagnosis, vision-language models, value of information

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LozoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents WIMW (Was It Me or the World?), a framework for scope-aware adaptation in embodied agents. The central problem is how an agent should determine the appropriate scope of a restriction after observing an execution failure, rather than repeatedly encountering the same limitation across subsequent tasks.The paper considers three possible explanations for failures: skill-wide capability limitations (Body), target-type-specific restrictions (World), and transient disruptions. Since an isolated failure does not necessarily distinguish these cases, directly generalizing such observations into persistent memory may lead to either overrestriction or underrestriction.

To address this problem, WIMW collects diagnostic evidence through target retries, cross-type reference tests, and final target verification. The resulting restrictions are encoded as capability contracts and enforced by an execution guard across tasks. The authors further introduce Patch-VOI, a decision-theoretic acquisition criterion that selects diagnostic actions based on their expected reduction in scope-decision loss while accounting for interaction and verification costs.The framework is evaluated with a frozen Qwen3-VL-32B planner on EB-ALFRED and EB-Habitat under injected pick failures. The experiments include comparisons with existing memory methods, executable-memory baselines, passive scope inference, and alternative diagnostic acquisition strategies.

Overall, the paper aims to move beyond passive failure memory toward active scope inference and enforceable cross-task behavioral restrictions.

Critical strengths

1: Meaningful problem formulation and a coherent motivation for executable failure memory

The strongest aspect of the paper is its formulation of failure memory as a scope-selection problem.Existing approaches to agent memory often emphasize how to summarize, store, and retrieve experiences. This paper asks a more specific question: how broadly should an observed failure affect future actions? This perspective highlights an important trade-off. Overly broad restrictions may suppress feasible actions, whereas overly narrow restrictions may allow the same underlying limitation to cause repeated failures across tasks.

The proposed capability contracts and execution guard provide a concrete mechanism for translating diagnostic conclusions into persistent behavioral constraints, rather than relying solely on a language model to follow textual advice.

I find this formulation practically meaningful, and the overall progression from scope uncertainty to diagnosis, verification, and execution-time enforcement provides a coherent research motivation.

2: Reasonably systematic evaluation with efforts to isolate the contributions of different components

The experimental study goes beyond a straightforward comparison against No update or general-purpose memory baselines.In particular, the authors introduce persistent executable caches and a matched Passive global + R baseline to investigate the additional value of reference acquisition while controlling for recovery, verification, execution guarding, and termination behavior.

Another positive aspect is that the paper distinguishes unexplained failures, task completion, total interactions, and scope-decision loss. The authors generally acknowledge that reducing failed attempts does not necessarily imply improved task completion.

Critical weaknesses

1.The main diagnostic mechanisms rely on relatively direct heuristic rules, while Patch-VOI is primarily built upon existing decision theory. More importantly, the connection between the theoretical methods and the main complete system experiments is not close enough, affecting the paper's technical innovation and overall consistency.

2.Current experiments mainly demonstrate that explicit behavioral constraints can reduce repeated failures under predefined fault conditions, but lack sufficient analysis of the correctness of scope inference, diagnostic decision-making mechanisms, and complex fault conditions. This means that the paper's core research claims have not been sufficiently validated.

Questions for authors
  1. Relationship between the reference controller and numerical Patch-VOI (clarification).

Could the authors clarify why the main full-stream experiments evaluate the rule-based reference controller, while numerical Patch-VOI is assessed separately under a different, shorter task-stream protocol? What is the intended methodological relationship between these two controllers, and what specific new insight does Patch-VOI provide beyond the reference diagnostic rules? In particular, would the advantages of numerical Patch-VOI remain when evaluated under the same full-stream execution protocol?

  1. Direct validation of scope-inference quality (clarification/additional analysis).

Could the authors provide a direct evaluation of the accuracy of inferred capability contracts, including false restrictions on feasible actions and failures to restrict genuinely unavailable actions? It would also be useful to analyze how different reference-test outcomes affect the final contract decision. More fundamentally, how does the proposed type-level restriction handle failures that differ across instances of the same object type? Under what assumptions is final verification on the triggering target sufficient to justify a type-wide restriction?

  1. Robustness and interpretability of active diagnostic gains (additional analysis).

Could the authors provide uncertainty estimates for the main paired comparisons in Table 3(b) and Table 4, together with a more fine-grained breakdown across fault conditions? How sensitive is numerical Patch-VOI to the relative penalties for overrestriction and underrestriction, as well as the diagnostic-cost weight? Under what conditions does Patch-VOI make substantially different or better decisions than fixed-order acquisition or simpler heuristic strategies?

4 comments
AuthorsOpenReview ↗

Part 1 of 2

We thank the reviewer for the careful reading and the specific questions. Our answers in brief are as follows. The reference controller and numerical Patch-VOI run the same diagnosis and differ only in how the next test is chosen, and Patch-VOI keeps its advantage when run in full streams. The contracts WIMW stores match the injected fault in every single-fault stream, and when only part of a type fails, the resulting error is a small over-restriction. The paired comparisons exclude zero wherever the paper reports a gain, and Patch-VOI's advantage holds at every cost weight we tried. The evidence follows, with results reported as EB-ALFRED / EB-Habitat.

Q1. Relationship between the two controllers, and Patch-VOI in full streams.

The two controllers are one procedure with two ways of choosing the next test. Under the full-stream protocol, Patch-VOI reaches the same contracts as the reference controller with fewer tests.

Both controllers retry the target, test reference objects, re-check the target before committing, and enforce the contract with the guard and periodic rechecks. The reference controller chooses the next test with two fixed rules taken from the analysis. A retry of the same target cannot separate a skill-wide fault from a type-local one (Prop. 1), so it tests other objects, and once a retry succeeds no further test can change the decision (Prop. 2), so it stops. Numerical Patch-VOI instead computes the choice from the expected reduction in scope loss minus the action cost (Eq. 6–7).

Patch-VOI adds a stopping rule tied to the stored decision. A test can reduce uncertainty about the fault and still leave the contract unchanged. Information gain keeps paying for such tests, whereas Patch-VOI skips them.

The paper evaluates Patch-VOI in 12-task blocks because a full stream gives the selector few decisions to make. WIMW commits a contract at the first unexplained failure, after which the guard blocks the failing action, so a stream with one persistent fault contains about one diagnosis. The blocks repeat this diagnosis with memory reset, which yields enough decisions to compare selectors.

In full streams with a single fault, Patch-VOI stores the same contracts with fewer tests. We placed it inside the deployed agent and reran the 60-task streams of Table 3(a), so both controllers share the planner, trigger, guard, recheck and step budget. Neither stores a wrong restriction. Patch-VOI needs 9.0 / 9.0 diagnostic actions against 11.0 / 10.5 under Body and 7.5 / 7.0 against 9.5 / 8.5 under World, and has 1.0 / 0.5 fewer unexplained failures under Body. Under Transient faults the two behave identically.

When a full stream contains repeated diagnoses, Patch-VOI still has the lowest cost. We built 300-task streams in which the fault changes every 60 tasks (Body, World on one type, Transient, World on a second type, Clean), ten per environment (two task orders × five executor seeds, Appendix B.4). Only test selection differs between arms, and JJ follows Eq. 13 with the paper weights, summed over the stream.

Switching streamsUnexplained failuresDiagnostic actionsUncovered demandsJJ
Reference (fixed order)89.6 / 89.388.6 / 88.24.0 / 3.140.86 / 36.42
Information gain89.9 / 89.4103.6 / 96.24.0 / 3.142.36 / 37.22
Numerical Patch-VOI88.7 / 88.671.6 / 77.22.6 / 2.237.76 / 34.42

All three selectors store a contract of the correct scope for all 60 persistent segments and block the same 5.6 / 4.9 feasible picks per stream. Those blocks come from contracts that remain in force after their segment ends, until a recheck lifts them. The selectors therefore differ only in the cost of diagnosis. Patch-VOI uses 17.0 / 11.0 fewer diagnostic actions than the fixed order and leaves fewer demands uncovered, lowering JJ by 7.6% / 5.5%. Information gain spends the most actions and has the highest JJ, the same ordering as in Table 4. The reduction is smaller than the 9.1% / 7.7% in the blocks because the shared over-restriction term makes up about 70% of JJ here. We will add both comparisons to Section 5.

AuthorsOpenReview ↗

Part 2 of 2

Q2. Correctness of the inferred contracts.

The contracts are correct. In every stream behind Table 3(a), WIMW stored exactly the contract the injected fault requires and blocked no feasible action. When instances of a type fail differently, the error is a small over-restriction.

We scored every stored contract and blocked action against the injected fault in the 40 streams behind Table 3(a). Each Body stream stored one skill-wide contract, each World stream one type-local contract on the faulty type, and Transient and Clean streams none. With the rates of Appendix A.2, WIMW leaves 3.1% / 2.6% of disabled demands uncovered, all before its first diagnosis, against 21.4% / 18.7% for the persistent cache. In the 12-task blocks of Table 4, where scope is ambiguous by design, Patch-VOI has the lowest rate of both errors. It blocks 1.2% / 1.0% of feasible demands, against 3.4% / 2.9% for the fixed order and 2.7% / 3.1% for information gain, and leaves 12.8% / 12.1% uncovered, against 14.2% / 13.7% and 15.3% / 13.9%.

The reference outcomes select the scope, and the target re-check decides whether anything is stored. References that also fail point to a skill-wide fault, and a reference success points to a type-local one. Under Transient faults, 80 diagnoses saw a reference succeed, and the re-check rejected all 80. Failures with a stated cause, such as "not near the robot", never count as evidence.

Verifying only the trigger target suffices when failures recur at the level at which actions are named. EmbodiedBench actions name object types, so the guard blocks a type, and the re-check, a fresh find and pick of that type, tests recurrence at that level. When we injected the fault on half or a quarter of a type's instances, type contracts formed in 6 of 11 and 3 of 13 diagnoses, against 4 of 4 when all instances fail, because the re-check passes whenever it reaches a working instance. A stored contract then blocks 3.5 / 2.5 and 1.5 / 0.5 feasible picks per stream, and WIMW still has fewer unexplained failures than no memory (7.25 against 15.5, 5.5 against 8.0). An interface that names instances would change only s(x)s(x).

Q3. Uncertainty and sensitivity.

The main comparisons hold under paired 95% intervals, and Patch-VOI's advantage holds at every cost weight we tried.

In Table 3(a), under Body, WIMW has 27.5 ± 0.7 unexplained failures against 72.5 ± 6.4 for the persistent cache in ALFRED, and 25.0 ± 0.0 against 58.5 ± 4.9 in Habitat. For Table 3(b) and Table 4, the paired differences in EB-ALFRED are below, with 95% bootstrap intervals over 16 paired streams. Negative values favor the first method.

Paired differenceBodyWorldTransientClean
ΔU\Delta U, matched WIMW − Passive global + R−3.9 [−5.1, −2.8]0.0 [0.0, 0.0]−0.1 [−0.6, 0.4]0.0
ΔJ\Delta J, Patch-VOI − fixed order−0.33 [−0.51, −0.16]−0.23 [−0.38, −0.09]−0.12 [−0.21, −0.04]0.00
ΔJ\Delta J, Patch-VOI − information gain−0.55 [−0.78, −0.33]−0.44 [−0.63, −0.26]−0.37 [−0.52, −0.23]0.00
ΔJ\Delta J, Patch-VOI − verification only−6.88 [−8.17, −5.64]−2.51 [−3.22, −1.83]−0.97 [−1.41, −0.56]0.00

The intervals exclude zero wherever the paper reports a gain, and reference acquisition helps only under Body, as Section 5 states. EB-Habitat has the same signs (condition means −0.13, −0.29 and −2.22), and Clean triggers no diagnosis.

We varied one weight at a time around the paper values (cover/cunder=5c_{\text{over}}/c_{\text{under}} = 5, λ=0.1\lambda = 0.1), and Patch-VOI stays ahead of the fixed order at every setting, with all intervals excluding zero. Its advantage grows with the over-restriction weight, from −0.06 at a ratio of 1 to −0.29 at 10, while its false-restriction rate falls from 2.7% to 0.6%. It also grows with the action cost, from −0.09 at λ=0.03\lambda = 0.03 to −0.34 at 0.3, because the fixed order keeps running tests that cannot change the contract. Patch-VOI therefore departs from the fixed order most when wrong contracts or tests are expensive: in 26.3% of diagnoses at the paper weights, 33.8% at a ratio of 10 and 38.1% at λ=0.3\lambda = 0.3. For the reference controller, varying each of its four thresholds left every contract in Table 3(a) unchanged. Failures moved by at most 1.5 per stream, except under the recheck frequency, which trades failures under a persistent fault against slower recovery once the fault clears.

We thank the reviewer again and would be grateful if these results lead to a reconsideration of the score.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to clarify the relationship between the reference controller and Patch-VOI and to strengthen the empirical evaluation. However, my main concerns regarding the generalizability of the approach remain. The predefined fault taxonomy closely matches WIMW's hypothesis space, making it unclear whether the method can generalize to more complex or unseen failure structures. The reliance on simulator metadata further limits its applicability to realistic embodied settings.

While I acknowledge the improved empirical evidence and diagnostic efficiency demonstrated in the rebuttal, these results primarily establish effectiveness within structured fault settings rather than generalizable failure-scope adaptation. Therefore, my concerns about the scope of the contribution and its applicability beyond the predefined fault assumptions remain, and I maintain my original rating.

We thank the reviewer for the follow-up and for recognizing the clarified relationship between the two controllers and the stronger evidence. The remaining concern, whether WIMW generalizes beyond faults that match its hypotheses, is a fair question for any method that reasons over a structured hypothesis space, and we would like to add two points that we hope are helpful.

The hypothesis space follows from what a memory can store, not from the faults we injected. A stored restriction has to block some set of future actions, and with an action interface that names skills and object types, that set is the whole skill, some object types, or nothing. Whatever the cause of a failure, whether joint wear, a slippery material or an occluding shelf, the memory has to choose among these sets. B, W_c and N are therefore the restrictions the agent can express, and the injected faults test whether it chooses among them correctly.

Several of the new conditions lie outside the three hypotheses, and in each WIMW either adapts correctly or errs on the safe side:

  • A limitation on a whole object class covers several types but not the whole skill, which none of B, W_c or N describes. WIMW restricted the affected types one at a time and never extended the restriction to the skill, reducing unexplained failures from 132.0 / 138.0 to 14.0 / 22.0.
  • A persistent fault overlaid with transient failures in every episode mixes two hypotheses. WIMW restricted only the persistently failing type, with 59.5 / 59.0 failures against 94.0 / 109.5 without memory and 54.0 / 54.0 for the oracle.
  • A fault on only some instances of a type cannot be expressed by any type-level contract. The re-check stores a contract less often as the faulty share drops, the over-restriction stays at most 3.5 / 2.5 feasible picks per stream, and WIMW still has fewer failures than no memory.
  • Faults that change during a 300-task stream were all covered by contracts of the correct scope.
  • The Clean streams contain 422–529 / 1,337–1,509 natural failures that we did not design. Even with the trigger filter removed, none of them produced a contract.

We also tested the diagnosis without simulator metadata. The agent read metadata for two signals, the objects present and the hand state. We replaced them with reference objects proposed by the same VLM from the camera image and with hand state tracked from action feedback. In EB-ALFRED the stored contracts were identical to those of the metadata version, and unexplained failures were unchanged (28.0 and 28.0 under Body, 3.0 and 3.0 under World). In EB-Habitat the agent can navigate only to receptacles, so without metadata it rarely reaches a second object to test. In that case WIMW stores no contract rather than a wrong one, which delays the restriction without causing over-restriction.

We agree that physical deployment and failure structures richer than skills and types are the natural next steps, and we will add a section on failures outside the hypothesis space, with the results above, to the revision. We hope these points clarify that WIMW does not depend on the fault matching its hypotheses, and that when it does not match, the error stays bounded. We would be grateful if the reviewer would take them into account in the final assessment, and we thank the reviewer again for the careful engagement throughout the discussion.

Reviewer_Yj2AOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses a question failure-memory systems rarely ask explicitly: after an unexplained failure, what is the scope of the restriction you should store? The authors formalise this as selecting a patch q over a capability contract m = (k, s, v) and define a scope loss L over a fixed reference workload w that penalises both over-covering and under-covering. This is an interesting, well-motivated paper with a genuinely novel framing.

Critical strengths

1.The problem formulation is the paper's real contribution, and it is genuinely good. Most failure-memory work stores a reflection or a principle; almost none asks over what set of future actions the lesson applies. 2.The results are strong where it matters and honestly framed elsewhere. Near-oracle failure counts under both Body and World indicate the mechanism is close to exhausting what can be extracted in this setting. 3.Every diagnostic action counts toward the interaction budget; releases, hand clearing, navigation, and picks are counted individually; adaptor metadata for reference types and hand state is declared as simulator-side rather than smuggled in through the oracle.

Critical weaknesses

1.The entire framework is about whether evidence is worth its interaction cost, yet the paper never closes that loop for its own headline comparison. A short sensitivity analysis over λ and (c_under, c_over), plus a direct "cost per avoided failure" accounting for Table 3(a), would either substantiate or overturn the practical claim. 2.All failures are injected at the skill interface and all return the same generic feedback; only PICK is ever restricted; scope is only ever object type. 3.Local contracts have no revocation rule, and the authors note that a temporary disruption surviving verification can produce a permanent restriction. 4.The X (unavailable) and E (explained) outcomes are modelled as preserving scope odds; a diagnostic budget exhausted without any informative test leaves will leave the agent with no contract.

Questions for authors

1.Why not run the numerical Patch-VOI selector through the 300-task streams, instead of only in the 12-task-block study? If the reference thresholds are hard-coded, please show sensitivity to alternative thresholds. 2.Can you run a small condition with naturally occurring planner failures (no injection) in Clean, to measure false-positive contract formation? This seems essential to the paper's applicability claims.

2 comments
AuthorsOpenReview ↗

Part 1 of 2

We thank the reviewer for the positive assessment and the precise questions. Results are reported as EB-ALFRED / EB-Habitat, and new runs use 60-task streams in two task orders unless stated otherwise.

W1. Is the evidence worth its cost?

Under Body faults in Table 3(a), WIMW has 45.0 fewer unexplained failures than the persistent cache in ALFRED and uses 257.5 more actions, which is 5.7 actions per avoided failure. In Habitat it has 33.5 fewer failures and also uses 112.0 fewer actions. Writing the realized cost as J=cunderU+coverO+λAJ = c_{\text{under}}U + c_{\text{over}}O + \lambda A, where OO counts blocked feasible actions, no method in Table 3(a) blocks a feasible action, so cover/cunderc_{\text{over}}/c_{\text{under}} cannot change any comparison there. WIMW then has the lower JJ for every λ<0.175\lambda < 0.175 in ALFRED, where the paper uses 0.1, and for every λ\lambda in Habitat. Under World faults it pays one extra failure per stream, which is the diagnosis itself, and 0.3% / 3% more actions.

Over-restriction does occur in the 12-task blocks of Table 4, so we swept the weights there for numerical Patch-VOI, varying cover/cunderc_{\text{over}}/c_{\text{under}} from 1 to 10 and λ\lambda from 0.03 to 0.3. Its JJ stays below that of fixed-order acquisition at every setting, with differences from −0.06 to −0.34 in ALFRED and every 95% interval excluding zero. The gap grows as a wrong contract or a test becomes more expensive.

W2. Narrow fault model.

We added conditions along each dimension the reviewer lists, all returning the same generic failure feedback as the original faults. When a persistent fault on one type is overlaid with a transient fault in every episode, WIMW has 59.5 / 59.0 unexplained failures, against 94.0 / 109.5 without memory and 54.0 / 54.0 for the oracle. When the robot cannot pick any member of an object class, it has 14.0 / 22.0 against 132.0 / 138.0. In 300-task streams where the fault changes every 60 tasks, it stores a contract of the correct scope for all 60 persistent segments. None of these conditions produced a wrong restriction, and removing simulator metadata from the diagnosis left every stored contract in ALFRED unchanged.

Scope below the type level is where a type contract can be too broad, so we also injected faults on only part of the instances of one type. Because the re-check passes whenever it reaches a working instance, type contracts form less often as the faulty share drops (6 of 11 diagnoses with half the instances faulty, 3 of 13 with a quarter). A stored contract then blocks 3.5 / 2.5 and 1.5 / 0.5 feasible picks per stream, while WIMW still has fewer unexplained failures than no memory. Faults remain injected at PICK. A contract is indexed by skill, so another skill changes which failures start a diagnosis, while the diagnosis and the guard stay the same.

W3. Revoking type-local contracts.

Skill-wide contracts already let every 4th blocked pick through as a recheck, and we now apply the same rule to type-local contracts. We tested it with a type-local fault that clears at task 30 in ALFRED and task 45 in Habitat. In Habitat, persistent contracts blocked 8.0 picks per stream after the fault cleared and completed none of the 9 later tasks on that type. With the recheck, 3.0 picks were blocked before the contract was lifted, and 4 of the 9 tasks were completed. In ALFRED the corresponding numbers are 10 blocks and 0 of 5 tasks against 6 blocks and 2 of 5 for one task order. In the other order the persistent variant formed no contract before the fault cleared, and the recheck variant lifted its contract after 3 blocks and completed 4 of 5. The rule costs 0.5 extra failure in Habitat, from a recheck made while the fault was still active, and in general one failure per four blocked picks under a persisting fault, the same price as the skill-wide recheck.

(Continued in the next comment.)

AuthorsOpenReview ↗

Part 2 of 2

W4. E and X outcomes, and an exhausted budget.

Explained failures such as "not visible" say nothing about the skill, so E/X leaves them out of the scope odds. Counting them would let navigation errors create restrictions. A diagnosis that ends without an informative test stores nothing, and the next unexplained failure starts a new one, so an exhausted budget can delay a contract but cannot create a wrong one. Under the persistent faults of Table 3(a), all 8 diagnoses reached a decision.

Q1. Patch-VOI in full streams, and threshold sensitivity.

Table 4 uses 12-task blocks because a stream with one persistent fault contains about one diagnosis. WIMW commits at the first unexplained failure, and the guard handles the rest of the stream. We have now run numerical Patch-VOI inside the deployed agent in full streams. On the 60-task streams of Table 3(a), it stores the same contracts as the reference controller with fewer diagnostic actions, 9.0 / 9.0 against 11.0 / 10.5 under Body and 7.5 / 7.0 against 9.5 / 8.5 under World. On 300-task switching streams (ten per environment), which contain repeated diagnoses, all selectors store the correct contract for every persistent segment. There, Patch-VOI lowers JJ by 7.6% / 5.5% relative to the fixed order, through 17.0 / 11.0 fewer diagnostic actions and fewer uncovered demands.

We also varied each threshold of the reference controller in turn, and every stream kept the contract of the paper setting with no wrong restriction. Requiring three failures instead of two for a skill-wide contract changes Body failures from 28.0 / 24.5 to 29.0 / 25.5. Requiring two reference successes for a type-local contract leaves World at 3.0 / 3.0. Halving the reference budget to three gives 29.5 / 23.5 under Body and leaves World unchanged, and no setting moves Transient failures by more than one. The recheck frequency is the one parameter with a large effect. Under a persistent Body fault every recheck is a failed pick, so rechecking every 2nd blocked pick gives 65.0 / 63.5 failures and every 8th gives 15.0 / 13.0. Frequent rechecks pay off when a fault clears, because a contract is lifted only by a successful recheck (W3).

Q2. Do natural failures create false contracts?

They do not. The Clean 300-task streams contain 422–529 / 1,337–1,509 failed picks, and WIMW formed no contract in any of the six streams. Most of these failures come with a stated reason, such as not visible or hand full, so the trigger skips them. To test the diagnosis itself, we removed this filter and diagnosed every failed pick. This gave 28.0 / 47.0 diagnoses per stream, of which 9.5 / 47.0 reached the commit re-check, and the re-check rejected all of them, so no contract was stored and no feasible action was blocked. The filter is still worth keeping, because diagnosing every failure uses up the step budget, and ALFRED then completes 19.0 tasks per stream instead of 26.0.

We hope these results answer the reviewer's questions.

Reviewer_ni5aOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper disentangles robot failures into two types: the "me" error from the robot itself and the "world" error from the environment.

Critical strengths

The problem itself is intriguing. It prompts us to rethink about failures, instead of havign the robot re-attempt over and over again. A good method for diagnosing and debuggign robotics failures.

Critical weaknesses
  • An overly simplified formulation of embodied failures. My main concern is that the paper substantially oversimplifies the diversity and complexity of embodied failures. Failures originating from the agent can involve joint malfunction, actuation limitations, perception errors, or planning failures, while environmental failures can involve unobserved physical properties, occlusions, dynamic changes, or complex physical interactions. However, the paper reduces this rich problem to a few predefined cases involving skill-wide, object-type-specific, or transient grasping failures. It is unclear how such a simplified formulation can generalize to realistic embodied settings, where failures often involve multiple interacting factors rather than a single predefined restriction.

  • An unclear and potentially misleading distinction between "me" and "world." Although the title raises an intriguing question, the definitions of "me" and "world" are not sufficiently justified. The paper essentially associates "me" with failures across object types and "world" with failures specific to an object type. However, the scope of a failure is not equivalent to its underlying cause. A robot's own limitations may manifest only when interacting with particular objects, while environmental factors may cause failures across many objects. More fundamentally, embodied actions arise from interactions between the agent and its environment, making these two sources of failure difficult, and sometimes impossible, to disentangle. Both "me" and "world" encompass far more than whether a robot can grasp one or multiple object types. The method therefore appears to classify the scope of observed failures rather than genuinely distinguish failures originating from the agent versus the environment.

  • Lack of real-world validation. All experiments are conducted in simulation with artificially injected PICK failures. The method also relies on simulator metadata for diagnostic actions. Without evaluating naturally occurring failures on physical robots, it is unclear whether the proposed diagnosis and adaptation framework generalizes beyond these controlled settings.

Questions for authors

See weakness

3 comments

We thank the reviewer for the thoughtful review. We agree with the central point of W2, and since it also answers W1, we begin there.

W2. The scope of a failure is not its cause.

We agree, and this distinction is the starting point of the method. WIMW does not try to find out why a failure happened. It decides what the agent should stop trying: the skill on every object, or the skill on one object type only. Two failures with different causes but the same scope call for the same memory, so separating their causes would not change what is stored (Section 3.1). In the paper, "me" and "world" are names for these two scopes, not claims about the cause.

The reviewer's two examples show why scope is the right target. A gripper that cannot hold one kind of object is a problem with the robot, yet the agent should avoid only that object type, which is what WIMW stores. An environmental condition that makes many objects fail comes from the world, yet the agent should stop using the skill everywhere until the condition clears, so WIMW stores a skill-wide restriction and rechecks it from time to time. In both cases a label based on the cause would produce the wrong memory, while a label based on scope produces the right one.

W1. The failure model is too simple.

WIMW models what a failure rules out rather than how it happened. Joint faults, actuation limits, perception errors, occlusion and unobserved physical properties all leave the agent with the same two questions: which actions no longer work, and for how long. Our conditions vary exactly these two factors.

The streams are also far from clean. Even without injection, every 300-task stream contains 422–529 (ALFRED) and 1,337–1,509 (Habitat) natural pick failures, from objects that are not found, occluded, too far away, or blocked by a full hand. None of them led WIMW to store a wrong restriction.

To test interacting failures directly, we added two harder conditions. The table reports unexplained failed picks per stream as EB-ALFRED / EB-Habitat (60-task streams, two task orders).

New conditionNo memoryWIMWOracleWrong restrictions (WIMW)
Persistent failure on one type plus a transient failure in every episode94.0 / 109.559.5 / 59.054.0 / 54.00
Robot limitation on a whole object class (the reviewer's example)132.0 / 138.014.0 / 22.00 / 00

In the first condition the transient failures do not mislead WIMW, and it restricts only the type that keeps failing. Even the oracle pays for the transient failures there, since no memory can prevent them. In the second condition, WIMW restricts the affected types one at a time and never extends the restriction to the whole skill.

W3. Real-world validation and reliance on simulator metadata.

We used controlled injection because our main question is whether a stored restriction is too broad or too narrow, and answering it requires knowing which actions really fail. Natural failures on a physical robot do not come with that ground truth, so injection is what makes the quantity measurable at all. We agree that a physical deployment is the next step (Section 6).

The controller reads metadata for only two things: which objects are present, used to choose reference objects, and whether the hand is holding something. We replaced both with signals a real robot has. The same VLM now proposes reference objects from the camera image, and hand state is tracked from action feedback. The evaluator still uses simulator state to inject faults and score the stored contracts, but the agent no longer reads any of it.

On EB-ALFRED (60-task streams, two task orders), the stored contracts stay exactly the same without metadata. WIMW stores one skill-wide contract under the robot-wide fault and one type-local contract when a single object type fails, and it stores nothing under transient faults or when there is no fault. Unexplained failures per stream are also unchanged (28.0 with metadata and 28.0 without, 3.0 and 3.0, 55.5 and 55.0, 0 and 0), and success and action counts stay within run-to-run noise. About 85% of the objects proposed by the VLM were actually in the scene.

EB-Habitat shows what happens when this information is missing. There the agent can only navigate to receptacles, so without metadata it rarely gets close enough to a second object to test it. In that case WIMW stores nothing instead of guessing, so missing information delays a restriction but does not create a wrong one.

To sum up, WIMW decides the scope of what an agent should stop doing, because a failure memory has to make that decision whatever the cause. The new results show that this decision holds when failures overlap, when a limitation covers only part of the object space, and when the agent has no access to simulator metadata. We hope this addresses the reviewer's concerns and would be grateful if the reviewer would consider raising the score.

I sincerely thank the authors for their thoughtful rebuttal and substantial additional experiments.

I appreciate the clarification that WIMW focuses on failure scope rather than underlying causes. The new experiments on overlapping failures and reduced reliance on simulator metadata also meaningfully address my concerns.

While the simplified failure formulation and lack of real-world validation remain limitations, I recognize the value of the proposed problem and the additional empirical evidence.

I am therefore willing to raise my rating to 3 (Weak Acceptance). Thank you again for your efforts to strengthen the work.

We thank the reviewer for reading our response carefully and for raising the rating. In the revision we will state the scope-versus-cause distinction explicitly in Section 3.1 and add the overlapping-failure conditions and the runs without simulator metadata to the experiments. Section 6 will discuss the simplified failure formulation and the lack of physical-robot validation as limitations, with deployment on a real robot as the next step. Thank you again for the constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.