acceptodds
Withdrawn from ICLR 2027

BELIEF BEYOND VISIBILITY: GROUNDING, UPDATING, AND PRESERVING PHYSICAL STATE FOR MANIPULATION

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Complex, long-horizon manipulation requires robots to update their estimates when interactions change an object's state and retain supported information when objects become hidden and viewpoints shift. Memory-augmented policies exploit interaction histories, while persistent scene models retain objects and relations across observations. However, task success and average attribute accuracy do not reveal when, after a transition, the system jointly recovers the required physical attributes of the correct task object. We formulate task-conditioned structured belief maintenance, including causal estimation of object existence, visibility, object–container relations, spatial evidence, and container articulation, with physical state separated from task roles. We introduce CERB (Control-Evidence-Routed Belief), which combines a learned multi-view object-centric observer with action-conditioned temporal inference and measurement-based belief revision. In CERB, state candidates are inferred from visual history, executed actions, and proprioceptive feedback, corrected by accumulated observation evidence, and integrated with the previously accepted belief by a learned update. Thus, new evidence can rectify an erroneous estimate without requiring a physical transition. To evaluate these capabilities, we introduce Primary35, a 35-task, three-view benchmark whose event-aligned criteria require applicable task binding and physical fields to recover together. Across three training repetitions, under the same frozen observer and complete output interface, CERB improves actionable recovery from to and complete-object recovery from to over the strongest adapted temporal baseline on in-distribution episodes held out from temporal training. The actionable-recovery gain persists when target-object assets are held out from temporal training.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotic learning, belief state estimation, object-centric learning, temporal reasoning, partial observability, structured prediction

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_vw5vOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents Control-Evidence-Routed Belief, a framework for maintaining structured physical-state beliefs in long-horizon robotic manipulation. The proposed method combines a learned multi-view object-centric observer with action-conditioned temporal inference and evidence-guided belief revision. It separates observability estimation, physical-state revision, and task-role inference into three branches, and maintains an explicit belief over object validity, visibility, spatial relations, geometry, and container articulation.

The authors also introduce Primary35, a 35-task simulation benchmark designed to evaluate joint belief recovery following physical-state transitions, transition timing, and hidden-state persistence. Experiments compare CERB against several adapted recurrent and state-space-model baselines under a shared frozen observer. The results demonstrate substantial improvements in actionable belief recovery and transition prediction, while a smaller downstream planning experiment provides preliminary evidence of the practical utility of the learned beliefs.

Critical strengths
  1. Relevant problem formulation. The paper addresses an important challenge in long-horizon robotic manipulation: maintaining accurate object-centric state estimates under occlusion, failed interactions, and changing task references. The distinction between physical-state estimation and task-role binding is conceptually meaningful.

  2. Structured evaluation framework. Primary35 introduces event-centered metrics that evaluate whether task-relevant physical attributes become jointly correct after state changes. These metrics provide useful complementary information beyond conventional frame-level accuracy.

  3. Systematic empirical analysis. The experiments compare multiple temporal estimation architectures under a shared observer and output interface. The authors also provide component analyses, uncertainty estimates, reference-sensitivity checks, and an initial downstream planning evaluation. The reported improvements in actionable belief recovery and transition prediction are encouraging.

Critical weaknesses
  1. Limited methodological novelty and excessive architectural complexity. The proposed framework combines several established components, including recurrent memory, local attention, object-centric representations, action-conditioned prediction, and learned evidence fusion. Although the integration is carefully designed, the individual mechanisms are largely conventional. The paper introduces a relatively complicated multi-branch architecture with specialized losses and training stages, but it remains unclear which aspects constitute a fundamental methodological advance over existing belief-state estimation and filtering approaches. A simpler formulation with comparable capabilities would make the contribution more convincing.

  2. Insufficient justification for the final architecture. A particularly concerning observation is that the complete CERB system does not consistently outperform its own alternative branch readouts. In Table 2, E+R achieves higher actionable recovery, complete-object recovery, and lower Event NLL than CERB, while P+R achieves better transition F1 on the in-distribution evaluation. Similar patterns appear under the target-asset shift. Although the authors discuss trade-offs among these metrics, the results raise questions about whether the proposed final field-assembly strategy is necessary or appropriately optimized. Stronger ablations would be needed to establish the benefits of the complete architecture over simpler alternatives.

  3. Potential confounding from unequal training budgets and model complexity. While the temporal baselines share the same frozen observer, input interface, and evaluation protocol, their training schedules differ considerably from CERB's. According to Appendix C, the baseline estimators receive 2,000 training updates, whereas CERB's physical-revision branch receives 3,320 cumulative updates, with additional branch-specific training and separately trained role inference. Consequently, the reported improvements cannot be attributed exclusively to the proposed belief-revision mechanisms. Comparisons with matched parameter counts, training compute, and optimization budgets would provide stronger evidence of the architectural contribution.

  4. Limited evidence of downstream manipulation benefits. The main improvements are measured using offline belief-estimation metrics rather than closed-loop manipulation performance. The downstream evaluation is restricted to 18 development starts across seven simulated scenes. CERB achieves 27/54 successful rollouts compared with 22/54 for the strongest competing downstream baseline, and the reported paired confidence interval includes zero. Furthermore, Appendix I indicates that CERB never explicitly declares task completion, even when the physical goal is achieved. These results provide only preliminary evidence that the proposed belief representation leads to more reliable robotic manipulation.

  5. Restricted evaluation setting and generalization claims. The evaluation relies on simulated manipulation streams with calibrated multi-view RGB-D observations, fixed-capacity object registers, and predefined physical-state categories. Moreover, the held-out target-asset evaluation excludes target assets from temporal training but does not establish fully independent generalization of the entire perception-and-belief pipeline. Although the authors provide sensitivity analyses for known observer-source overlap, these do not establish independence from all observer-training sources. Therefore, the extent to which the proposed approach generalizes to real-world perception noise, unseen environments, and dynamically changing object inventories remains uncertain.

  6. Limited gains under more stringent measures of belief quality. The primary ABR@8 metric requires the system to produce a jointly correct estimate at least once within a post-transition window, which does not necessarily imply persistent correctness. Although the authors provide complementary timing and retention diagnostics, the improvements are considerably smaller for complete-object recovery and sustained recovery. Additionally, the temporal diagnostics reveal substantial rates of premature visibility and articulation predictions. These observations suggest that the improvements in actionable recovery do not uniformly translate into stable and temporally accurate physical-state estimation.

Questions for authors
  1. Necessity of the complete architecture. In Table 2, both E+R and P+R outperform the complete CERB system on several key metrics. Could the authors explain more clearly why the proposed final architecture should be preferred over these alternative readouts? Is there a downstream evaluation demonstrating a consistent advantage of the complete system over its individual branches?

  2. Fairness of computational comparisons. Appendix C reports substantially different training schedules for CERB and the competing temporal estimators. Could the authors provide comparisons with matched training budgets, parameter counts, and computational costs? How much of the reported improvement can be attributed specifically to the evidence-guided revision mechanism rather than increased architectural capacity or additional optimization?

  3. Comparison with simpler belief-update mechanisms. Could the authors compare the proposed learned reliability-based update against simpler alternatives, such as fixed-weight belief fusion, exponential smoothing, or conventional Bayesian filtering with learned observation and transition models? Such comparisons would help clarify whether the proposed update mechanism offers advantages beyond standard state-estimation techniques.

  4. Downstream task performance and completion recognition. The downstream evaluation shows a relatively small improvement over SlotSSM, with a confidence interval that includes zero. Furthermore, Appendix I reports that CERB never outputs DONE despite achieving physical task success in some rollouts. How do the authors interpret this discrepancy? Could the approach be evaluated on a larger set of independently sampled closed-loop manipulation tasks, including explicit completion recognition?

  5. Robustness and real-world generalization. How sensitive is CERB to noisy depth measurements, incorrect object associations, missed detections, and changes in the number of objects or containers? Could the authors clarify how the framework would extend beyond fixed-capacity object registers and the predefined relation vocabulary used in Primary35?

  6. Temporal correctness and belief persistence. Given that ABR@8 requires only one correct prediction within the recovery window, could the authors further explain how the system balances rapid correction against sustained consistency? In particular, the premature articulation predictions reported in Appendix H appear relatively frequent. What mechanisms could reduce these premature predictions without sacrificing recovery performance?

2 comments

We thank the reviewer for the detailed comments and respond to each point below.

W1. Novelty and complexity

CERB is not an assembly of standard parts. Its contribution is a different way of revising a belief, and three properties follow from it that a single recurrent estimator does not provide by construction:

  • An attempted action is not an outcome. The model separates whether evidence is available from which state it supports, so a closed gripper does not write HELD into the belief.
  • The belief can be corrected without any physical change. A failed grasp is evidence against an earlier HELD estimate even though nothing moved.
  • Nothing is rewritten without evidence, and task queries never alter the physical state. A hidden object's state is copied exactly (Eq. 49), and a new query changes only the role answer (Eq. 2).

Recurrent memory and local attention only help form the candidate that this rule revises. The structure is also simple to state: Stage I reads the scene, Stage II revises the belief, and three branches answer three questions (what is observable, what changed, which object the task refers to). As for a simpler formulation with comparable capability, Table 1 is that test. It keeps our visual front-end and replaces Stage II with a single temporal estimator, and ABR@8 falls from 59.60% to at most 38.90% (IID) and from 61.44% to at most 49.10% (target assets).

Q1 (W2). Why the complete system rather than E+R or P+R

The assembly follows the design of the system, not a search over readouts. Each public field is read from the branch built to estimate it: validity and visibility from E, relation and articulation from P, roles from R. Table 2 reports E+R and P+R precisely to show what each branch contributes.

The practical criterion is the planner. Its decisions are driven by relation and articulation: App. I shows that swapping only the articulation probability flips its choice between OPEN and GRASP, while swapping visibility does not. These are the fields where P is far better timed than E (relation F1 .469 against .195). E+R gains 1–3 points of recovery within the window, but it hands the planner poorly timed relation updates. P+R keeps the timing but is 2.02 / 1.46 points lower on Whole@8. The complete system keeps both properties.

Q2 (W3). Training budget and capacity

Table 1 compares complete belief systems, as Sec. 5.1 states, and we do not attribute it to one mechanism. Capacity is comparable: CERB's Stage II has 13.1M parameters and the Gated DeltaNet-2 system has 12.4M. The role branch R is the SlotSSM system itself, trained for the same 2,000 updates. The counts in Table 4 follow one training path each and are not additive, since E and P share their first 1,600 updates.

What the revision mechanism itself contributes is measured where the budget is identical by construction:

  • In the same trained model, the belief after the revision step is better than the candidate before it on all four metrics on IID (Whole@8 by 1.5 points, Transition F1 by .027), and Transition F1 is .057 higher on target assets.
  • Removing the revision path and retraining with the same schedule lowers Transition F1 by .026 / .036, mainly on relation changes.
  • CERB and the SlotSSM system share the same role readout, so their difference comes from physical-state estimation.

Q3. Simpler update rules

Fixed-weight fusion and exponential smoothing are the same special case of Eq. 6: constant weights on the previous belief and the new candidates. We compared against exactly this, with the constant weights learned. The reliability-based update assigns higher probability to the true state: Event NLL is lower by .023 (IID) and .037 (target assets). A constant weight cannot do what the task needs, because it treats the action candidate after a failed grasp the same as after a successful one.

A conventional Bayesian filter combines an observation model and a transition model under an independence assumption. Here the candidates are computed from the same visual history and are strongly correlated, so we learn a discriminative update over the named fields instead (Sec. 2, App. J).

Q4 (W4). Downstream performance and DONE

The subject of the paper is belief estimation, and the Introduction explains why task success alone cannot evaluate it: success also depends on the planner and the controller. The planner study shows that the belief is usable, and we present it as a proof of concept. Within it, CERB is the best of the five systems (27/54 against 22, 17, 13 and 7) and best or tied in every start type.

On DONE: success is judged by an independent physical check, not by the DONE signal. The planner declares DONE only after a VERIFY and with goal probability of at least .8, and CERB's belief did not meet these conditions within the budget. It made no completion claim and therefore no false one. The other systems declared DONE 13 times, and 11 of those declarations were wrong (Table 20). Recognising completion is a separate capability, as Sec. 6 states.

Q5 (W5). Robustness and scope

  • Association errors and missed detections. Every result already includes them. The observer is learned and imperfect (82.93% instance accuracy), and all numbers are obtained with its outputs, not with ground-truth perception.
  • Noisy or missing depth. Each measurement carries its own support. Without depth support the geometry is not written, and the last supported estimate is kept.
  • Number of objects and containers. The front-end was extended from four to five container registers by reusing 587 of its 600 parameter tensors, with no further alignment training (App. B.5).
  • Relation vocabulary. Relations are an explicit list, which is what makes the belief checkable and usable by a planner. A new relation needs a new output state; the revision rule does not change.
  • Scope of the claim. We claim temporal generalization under a shared frozen observer, as stated in Sec. 3.2. With every episode from a known observer source removed, the advantage remains (ABR@8 62.15% against at most 41.88% on IID; 59.49% against at most 51.96% on target assets; Table 17).

Q6 (W6). Temporal correctness and persistence

ABR@8 and Transition F1 are separate metrics on purpose. ABR asks whether the full state is right within the window, and Transition F1 asks whether the belief changed at the right moment. CERB leads on both, and on how stably it holds a relation:

  • Transition F1 is .305 against at most .170 (IID) and .458 against at most .276 (target assets).
  • A hidden object's relation is still correct eight steps into occlusion in 73.47% / 93.37% of cases, against at most 62.86% / 85.12% (Table 14).
  • Relation beliefs change while the reference is stable less often than in any alternative (3.33% / 2.91%, Table 15), and CERB predicts a relation's destination early less often than any alternative (12.87% / 9.90%, Table 16).

The balance between fast correction and consistency comes from the update itself: with no evidence the belief is copied, and with evidence the learned reliability decides how far it moves.

The early articulation predictions have a specific form. CERB switches its belief when the door starts to move, not when it reaches the terminal threshold; about 98% of these predictions fall while the door is between 20% and 80% open. This is not a property of the revision rule: relations, which the same rule updates, show the opposite behaviour. A direct remedy is to supervise the opening fraction as a continuous quantity, so that the belief switches at the threshold.

Reviewer_74DtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies structured physical-state belief maintenance for robotic manipulation under partial observability. It introduces PRIMARY35, a simulation benchmark with event-centered metrics for jointly recovering task-object bindings and applicable physical attributes. It also proposes CERB which combines a learned multi-view observer with separate temporal branches for observability, physical-state revision, and task-role inference. Physical beliefs are updated through measurement-based correction and learned pooling of previous beliefs and current candidates. Under a shared frozen observer, CERB improves actionable belief recovery and transition F1 over adapted temporal baselines, while improvements in complete-object recovery are considerably smaller.

Critical strengths
  1. A well motivated problem formulation. Separating physical state from task roles, and preserving supported state information through occlusion, are important capabilities for long-horizon manipulation.

  2. The evaluation goes beyond average frame-level accuracy. Joint recovery, transition timing, and hidden-state retention capture complementary aspects of belief quality that conventional attribute accuracy can miss.

  3. The experimental analysis includes useful controls and qualifications. The shared observer helps control perceptual differences between temporal systems. The paper also reports three training repetitions, paired uncertainty estimates, reference-sensitivity analyses, and limitations of the downstream study.

Critical weaknesses
  1. Poor presentation makes the method difficult to reconstruct and it is substantive rather than stylistic. The main text cannot be read without the appendix. The paper's key object (the belief schema) is compressed into a single paragraph and only fully specified in the Appendix. Section 4 introduces multiple interacting mechanisms, however, essential information is distributed across Sections 4.2–4.4 and Appendices B and C. Figure 1 provides a useful overview, but does not sufficiently explain the distinctions among these overlapping temporal and correction pathways. This substantially increases the effort required to assess the technical contribution.

  2. The reported readout comparisons do not establish why the final architecture is preferable. In Table 2, E + R outperforms CERB on ABR@8, Whole@8, and Event NLL on both test cohorts. On IID episodes, P + R also achieves higher ABR@8 and transition F1 than CERB. The authors acknowledge these trade-offs, but the justification for the final assembly remains incomplete. In particular, downstream comparisons against E + R and P + R would help establish whether the additional branch combination provides practical value.

  3. The headline recovery results require a more prominent discussion of premature state predictions. ABR counts a recovery whenever the estimate becomes jointly correct at least once within the post-event window; it does not require a correctly timed update. This distinction matters substantially here. Table 14 reports 100% IID container recovery for CERB, while Table 16 shows an articulation early-destination prediction rate of 83.67%, compared with 17.20% for Gated DeltaNet-2. These results are disclosed, but their implications deserve greater emphasis in the main experimental narrative. High container recovery alone provides limited evidence of timely belief revision.

  4. The ablations do not sufficiently isolate the central proposed update mechanism. Table 2 compares readouts of already trained branches, rather than independently removing the main components of physical revision. Appendix G studies proposal components on development streams under a different input condition with supplied candidates. Consequently, it remains unclear how much of the final-system improvement comes specifically from measurement correction, accepted-posterior recursion, learned reliability pooling, or command–outcome balancing. These mechanisms are central to the claimed contribution and warrant direct ablations under the main evaluation protocol.

Questions for authors
  1. Could you provide a single-step pseudocode algorithm showing all recurrent states, candidate computations, correction operations, and public-belief updates?
  2. What practical criterion motivates selecting the final CERB assembly over E + R or P + R? How do these alternatives perform with the shared downstream planner?
  3. How much of the container-recovery advantage remains when recovery requires a correctly timed transition and sustained correctness afterward?
  4. Can you independently ablate measurement correction, accepted-posterior recursion, learned pooling, and command–outcome balancing in the final system under the main inputconditions?
3 comments

We thank the reviewer for the detailed reading and respond to each point below.

Q1 (W1). Single-step algorithm

The complete update is in the main text (Eqs. 5–7). We agree that a single-step view makes it easier to follow, and we will add the one below to Sec. 4 together with the belief schema as a table.

Carried from one step to the next: the public belief (the state exposed to the robot), a private history for each branch, and the accumulated measurement evidence.

One step, when a new observation arrives:

  1. Ground the evidence (Stage I). The three views and the instruction are turned into per-object measurements. Each measurement carries its own support, meaning how much the current frame directly backs it. A prediction without support is never treated as an observation.
  2. Read the completed action. The actions executed since the last observation are read together with the proprioceptive response that followed them. An attempted action is kept separate from its outcome.
  3. Propose (Eq. 5). From its own history, each physical branch proposes the current state of every field. For relation and articulation it also forms a second candidate: the state the executed action would lead to.
  4. Correct with measurements (Eq. 7). The proposal is adjusted by the accumulated measurement evidence. This path sees what was measured, not what was commanded, so a plausible but wrong proposal can be overturned.
  5. Revise the public belief (Eq. 6). Three alternatives are weighed by a learned reliability: keep the previous belief, adopt the corrected candidate, or adopt the action candidate. With no new evidence, the previous belief is copied exactly. The accepted belief becomes the prior of the next step.
  6. Assemble the remaining fields. Visibility and validity come from E, and geometry from P. Each is written only when its own evidence supports it; otherwise it is retained.
  7. Answer the task query. R decides which objects the task refers to. The query never feeds back into steps 1–6, so it cannot change the physical belief.

What is new lies in steps 1, 4, 5 and 7. Evidence is separated from prediction, measurements can overturn an action-based expectation, the revision acts on the belief the robot actually uses and recurs on it, and task roles are decoupled from physical state. A recurrent estimator that maps history directly to an output has none of these by construction.

Q2 (W2). Why the final assembly rather than E+R or P+R

The assembly follows the design of the system, not a search over readouts. Each public field is read from the branch built to estimate it: validity and visibility from E, relation and articulation from P, roles from R. Table 2 reports E+R and P+R precisely to show what each branch contributes.

The practical criterion is the planner. Its decisions are driven by relation and articulation: App. I shows that swapping only the articulation probability flips its choice between OPEN and GRASP, while swapping visibility does not. These are the fields where P is far better timed than E (relation F1 .469 against .195). E+R gains 1–3 points of recovery within the window, but it hands the planner poorly timed relation updates. P+R keeps the timing but is 2.02 / 1.46 points lower on Whole@8. The final assembly keeps both properties, and it is the belief used in the planner study of Sec. 5.3.

Q3 (W3). Recovery and timing

ABR@8 and Transition F1 are separate metrics on purpose. ABR asks whether the full state is right within the window. Transition F1 asks whether the belief changed at the right moment. We report both in Table 1 so that recovery is never read as timing, and CERB leads on both: Transition F1 is .305 against at most .170 (IID) and .458 against at most .276 (target assets).

On containers, CERB switches its articulation belief when the door starts to move, not when it reaches the terminal threshold: about 98% of its early predictions fall while the door is between 20% and 80% open. That is why container ABR is high, and we do not present it as evidence of articulation timing. Our conclusion does not rest on it:

  • On object events alone, CERB is ahead of every alternative (Object ABR 43.86% against at most 39.06% on IID; 56.39% against at most 53.90% on target assets; Table 14).
  • When we additionally require three consecutive correct observations after the change, complete-object recovery stays ahead of the strongest alternative in Table 1 (43.00% vs 39.52%; 49.44% vs 46.19%).
  • For relations, the field the revision targets, CERB predicts the destination early less often than any alternative (12.87% / 9.90%; Table 16).

Q4 (W4). Ablation of the update mechanism

We see this differently. CERB is a two-stage system, and no existing method produces its complete belief output, so we ablate it level by level.

  • Stage II as a whole. Table 1 keeps our visual front-end fixed and replaces Stage II with a single temporal estimator. ABR@8 drops from 59.60% to at most 38.90% (IID) and from 61.44% to at most 49.10% (target assets).
  • Branches. Table 2 shows that R improves task binding and P improves transition timing.
  • The revision step. Measurement correction, reliability pooling and recursion on the accepted belief are not independent add-ons. The correction produces the candidate that the pooling weighs, and the pooled result is what recurs. We therefore test them as one path. Reading the candidate before this step, instead of the belief after it, is worse on all four metrics on IID (Whole@8 by 1.5 points, Transition F1 by .027), and Transition F1 is .057 lower on target assets. Removing the path and retraining lowers Transition F1 by .026 / .036, mainly on relation changes. Replacing learned reliability with an input-independent mix worsens Event NLL by .023 / .037.

This is the role the path was designed for: getting the moment and the confidence of a change right, which ABR@8 by definition does not measure. Command–outcome balancing is a training term, not an inference mechanism. It stops an attempted grasp from being learned as a successful one (Sec. 4.4).

The rebuttal clarifies CERB’s single-step belief update, and the retrained ablation shows that the revision path improves transition timing. My main concern remains that the manuscript needs the belief schema and pseudocode in the main text so readers can reconstruct the method. The downstream evaluation still lacks a comparison with E+R and P+R using the same planner, and it remains unclear how much of the container-recovery gain holds when both correct timing and sustained correctness are required. Given these concerns, I maintain my rating of 2 (Weak rejection).

Reviewer_q5t6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces CERB, a memory system that supports queries about object attributes and task-roles. It grounds object states and their probabilities through an encoder built on top of a frozen SigLIP2 model into a fixed schema and updates states and their probabilities with a recurrent network given new actions and observations. The system is validated on a benchmark they introduce on whether the memory is correctly updated.

Critical strengths
  1. Maintaining persistent object-state memory across interaction and occlusion for long-horizon manipulation is an important and timely problem.
Critical weaknesses
  1. Writing was unclear to me. New topics and a myriad of terms are introduced without any explanation, often in sentences that have no clear connection to the previous.
  2. Overall: the contribution is unclear to me. Learning object-level state and relations from perception (stated contribution 2) and updating estimates over time under occlusion and interaction (stated contribution 3) has been studied, e.g. RoboEXP (Jiang et al., CoRL 2024), DynaMem (Liu et al., 2024), among others. Some of these are cited but not adequately compared / distinguished against both conceptually and experimentally. The benchmark (stated contribution 1) is helpful, but only variations of the method are tested.
  3. Method: the memory seems to be a fixed-sized state dictionary over predefined attributes, covering 6 objects and 5 containers, which does not very useful. State estimation already relies on foundation models (SigLIP2), so it's unclear why one would force a particular fixed-size representation and run a recurrent network over it. Again the writing is obscuring what the method actually is; for example, the paper introduces things like "causal, task-conditioned physical belief" and "public belief", but from my understanding it all seems to be just state-classification probabilities.
  4. Experiments: Rather than comparing against different memory systems (e.g. RoboEXP), the experiments compare CERB against other trained networks for memory updates within the same framework. So nothing isolates why this particular fixed-size memory and design is even helpful. For example, the instinctual baseline to me would be using another foundation model over a variable size state representation. Other soundness issues in the experiments: e.g., CERB combines three branches (E, P, R), with P trained for 3,320 updates and E for 2,280, vs. 2,000 for each baseline -- compute isn't matched, so not really a fair comparison.
Questions for authors
  1. What is the specific contribution over existing systems that track object states and update their estimated with new observations? Which experiment isolates that contribution?
2 comments
AuthorsOpenReview ↗

Clarify our contributions and address some misunderstandings

We thank the reviewer. CERB is an online, task-conditioned physical-belief estimator system architecture, not a scene-memory framework. We clarify its scope and contributions.

1. What “belief” means

Three examples: (i) A jar stays in a refrigerator after the door closes, despite invisibility. (ii) After a failed grasp, evidence must correct an erroneous HELD estimate although the jar did not move. (iii) A new task query can reassign roles without changing physical state. These require updating after changes, correcting errors without changes, and revising task roles independently.

  • Causal belief: uses only observations, executed actions, and proprioceptive feedback available up to the current time (Sec. 3.1; Eq. 1).
  • Physical belief: distributions over object validity, visibility, relation–anchor states, geometry, and container articulation (Sec. 3.1; App. B.4).
  • Accepted/public belief: the emitted distribution, available to queries and serving as the prior for subsequent revision (Sec. 4.3; Eq. 6).

CERB recursively estimates state, rather than classifying frames independently.

2. Contributions and relation to RoboEXP/DynaMem

We appreciate the reviewer's references to RoboEXP and DynaMem. However, instead of building a general memory dictionary or reconstructing scenarios, we focus on online, task-conditional estimation of physical beliefs. CERB represents a distinct paradigm: **We formulate and address online, task-conditioned physical-belief estimation for manipulation: estimating what is physically true now, how new observations and executed actions should revise that estimate, and which information should be retained when direct evidence becomes unavailable.Our approach is completely different from that of the other two papers, and the problems we address are also distinct.

Contributions:

(1) Evidence-guided online belief revision. CERB estimates realized, not commanded, outcomes from visual history, executed actions, proprioception, and accepted beliefs. Its E/P/R branches separately estimate observability (E), revise physical state (P), and answer task-role queries without altering physical estimates (R). Distinct temporal representations undergo field-specific assembly. Within P, measurement evidence corrects temporal candidates; a learned reliability mechanism integrates the corrected estimate, action-conditioned candidate, and prior accepted belief (Eqs. 5–7). The emitted distribution becomes the next prior. Physical change ≠ belief revision: failed actions can prompt correction without movement; occlusion need not invalidate supported state.

(2) Learned multi-view object-centric observer. Beyond frozen SigLIP 2, Stage I learns instruction-conditioned entity proposals, calibrated three-view RGB-D fusion, concept-level foreground estimation, competitive instance association, and typed physical measurements with field-specific support (Sec. 4.1; App. A). Separating foreground evidence from competitive instance allocation avoids duplicating evidence across slots. Conditional instance accuracy increases 69.47%→82.93%, and same-category accuracy 35.30%→49.59% (App. C.1). The Stage-II-frozen observer gives temporal estimators identical measurements and support.

(3) Primary35 benchmark. Primary35's 35 tasks assess joint post-interaction recovery of task-object binding and applicable physical fields, beyond frame accuracy or policy success. ABR@8/Whole@8 measure joint recovery; Transition F1, Event NLL, and hidden-state diagnostics assess transitions, probability quality, and retention (Sec. 3; App. D). This isolates belief quality from downstream planners.

Difference:

DimensionRoboEXPDynaMemCERB (Ours)
Primary objectiveInteractive exploration and construction of an action-conditioned scene graphDynamic object localization and search for open-vocabulary mobile manipulationOnline estimation and revision of task-relevant physical belief
Core representationExplicit action-conditioned scene graph with low-level 3D and high-level relational memoryDynamic 3D voxelized point-cloud and semantic-feature memoryObject-centric probability distributions over typed physical-state fields
Update mechanismFuse newly explored scene information and modify graph nodes, edges, and spatial memoryAdd, update, and remove spatial memory points as the environment changesInfer state from observation/action history, correct candidates with physical evidence, and recursively update the accepted belief
Primary evaluationScene exploration, graph construction, object discovery, and downstream manipulationDynamic 3D object localization, search, and mobile-manipulation successJoint post-change physical-state recovery, transition timing, probability quality, and hidden-state retention
AuthorsOpenReview ↗

Clarification of the methods and experiment sections

3. Method: competitive object slots and coordinated E+P+R.

Object slots:

The six object and five container slots are not object dictionary. The object queries compete to explain the observed evidence. This mechanism is learned.(Sec. 4.1; App. A.1; Fig. A1). To distinguish instances, including objects of the same category, CERB first detects task-relevant foreground and then learns which object query owns each patch (Sec. 4.1, Eq. 4; App. A.3, Eqs. 11–12):

\pi_{in}=\frac{e^{\ell_{in}+r_{in}}}{\sum_{j\in\mathcal R_o}e^{\ell_{jn}+r_{jn}}},$$ $$\Gamma_{in}=\gamma_n\pi_{in},\qquad \sum_{i\in\mathcal R_o}\Gamma_{in}=\gamma_n.$$ Here $p_{cn}$ is patch $n$’s probability for concept $c$; $\gamma_n$ is the combined foreground probability; and $\pi_{in}$ assigns it to query $i$ using learned scores $\ell_{in}$ and corrections $r_{in}$. $\mathcal C_o$ and $\mathcal R_o$ are the active concepts and available queries. This is learned instance association, not dictionary. It improves conditional instance accuracy from **69.47% to 82.93%**, and same-category accuracy from **35.30% to 49.59%** (App. C.1). **E+P+R:** E/P/R serve distinct but coordinated roles. The specialization is supported experimentally (Sec. 5.2; Appendix F): - Compared with E+R, CERB improves Transition F1 from 0.140 to 0.305 on IID and from 0.178 to 0.458 on target assets. - Compared with P+R, CERB improves Whole@8 by 2.02 and 1.46 percentage points, respectively. - Dedicated R also raises IID role-set exactness to 89.62%, compared with 74.95% and 81.06% for the native E/P role. These results demonstrate complementary capabilities. **E+P+R is designed to jointly provide observability, physical-transition estimation, complete-state recovery, and task binding—not to maximize one isolated metric.** ## 4. Experiments **Comparisons Between Belief-Estimation Systems:** We would like to clarify the experimental design in Sec. 5.1, particularly the distinction between **comparing complete belief-estimation systems and comparing individual neural operators under identical training budgets**. GRU, SlotSSM, Gated DeltaNet-2, and Mamba-3 are four distinct temporal models. **However, as they were not designed for our specific robotic task—and no off-the-shelf, ready-to-use checkpoints are available—we trained each of them on our own data and then compared their performance.** All methods use the same trained and fixed multi-view observer (This is one of our contributions). This unified interface is a deliberately designed experimental control method aimed at isolating the "time-based belief estimation capability" to avoid confusion with differences in visual perception. We further investigate different aspects of the proposed system: - **Table 1:** Comparisons among five complete belief-estimation systems. - **Table 2 and Appendix F:** Functional specialization and complementary contributions of E/P/R. - **Appendix G:** Contributions of local attention and typed change memory. - **Appendix C.1:** Effectiveness of learned visual-instance association. These experiments evaluate the complete system at multiple levels rather than simply comparing variations of a single recurrent model. **Training Budgets and Comparison Fairness:** **Regarding training budgets, different numbers of optimization updates do not inherently imply an unfair comparison.** The reported update counts correspond to the selected training configurations and retained checkpoints of the respective systems, rather than a universal stopping-step requirement. **This principle applies to CERB and all four baselines alike.** The different update counts should therefore not be interpreted as an arbitrary advantage assigned to CERB. More fundamentally, different architectures have different convergence characteristics, optimization difficulties, parameterizations, and computational costs per update. **Forcing GRU-, Transformer-, and state-space-based systems to undergo an identical number of optimizer steps does not necessarily equalize their computational budgets or provide an equally appropriate optimization procedure**. Under this controlled evaluation, CERB improves ABR@8 from **38.90% to 59.60% on IID** and from **49.10% to 61.44% on target assets** relative to the strongest sequence baseline. It also achieves substantially higher Transition F1 and lower Event NLL (Table 1). These results demonstrate the effectiveness of CERB as a complete system for online, task-conditioned physical-belief estimation. The differences in optimization-step counts alone do not invalidate these complete-system performance comparisons.
Reviewer_L4McOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies physical state maintenance in robot manipulation. The authors propose a two-staged method, CERB (Control-Evidence-Routed Belief), which combines multi-view object observations, actions, temporal memory, and evidence-based state correction. In the first stage, a multi-view, object-centric observer is trained. In the second stage, temporal inference are splited into three branches based on visibilities, physical-field revision, and task-role readout. The information is further fused as a physical attribute for each step. To evaluate the ability of physical understanding and preservation, the paper introduces PRIMARY35, a 35-task benchmark that uses event-aligned metrics to measure whether a system recovers the correct task-relevant state after a state transition. The ABR@8 and Whole@8 metrics measure whether role binding and all physical fields become jointly correct within a finite event window. CERB achieves progress in these metrics, demonstrating the promising abilities of the architecture.

Critical strengths
  1. The problem definition is valuable. The authors measure belief quality as event-aligned, joint correctness within a finite recovery window for memory and state-estimation research.
  2. The experiment is fair. All baselines share the frozen observer, inputs, and evaluation settings. Three training repetitions give means and standard deviations. The appendix includes several sensitivity analyses and honestly reports cases where confidence intervals include zero.
  3. The improvements on the main metric ABR@8, as well as on transition F1 and Event NLL, are large and consistent across both test suites, and the role-inference branch R clearly improves role-set exactness.
Critical weaknesses
  1. The system architecture is complex, making the source of the gains unclear. CERB simultaneously uses multiple branches, temporal encoding paths, field-level memory, action-conditioned candidates, and evidence-based state correction. The existing analyses provide some clues, but it is still hard to see which parts of the added complexity contribute to the final results.

  2. The authors' readout comparison shows limitations. In Table 2, the settings of E+R achieve higher recovery than full CERB, but CERB has better transition-timing F1. This shows that different readouts matter. As a result, the full readout is not always better than simpler alternatives.

  3. The connection between downstream utility and the state-estimation metrics is not clear. Consistent improvements in task success and recovery under the same policy and manipulation environment will demonstrate the capability of this structured belief system.

  4. The method depends on object registration, observer quality, and predefined state fields and relations. Unknown objects, detection errors, and new types of physical relations in open scenes may bring scaling costs and error propagation.

Questions for authors
  1. Can the author provide a component-wise ablation on the main benchmark to verify the effectiveness of each component?
  2. In Table 2, E+R achieves higher recovery than the full CERB, while CERB achieves better transition-timing F1. What explains this trade-off? Does adding the P branch introduce errors during recovery under certain perturbations?
  3. What is the correlation, across methods and tasks, between the state-estimation metrics and downstream success? Which metric best predicts downstream performance?
  4. How does CERB handle objects that were not registered or proposed at the start of an episode, such as objects entering the scene mid-episode or objects not mentioned in the instruction?
  5. For scenes containing object categories or relation types unseen during training, how will the design scale to these open-world settings?
2 comments

Thank you for the careful review. We think most of the concerns come from how the paper is organised, so we first restate its structure and then answer each question.

CERB has two stages, and both are our contributions. Stage I is a learned multi-view visual front-end. It turns the images and the instruction into per-object measurements, each with a score for how much evidence supports it. Stage II maintains the belief over time from those measurements, the executed actions and proprioception.

Q1 (W1). Component ablations and the source of the gain

No existing method produces this complete belief output, so no off-the-shelf system can be compared directly. Our experiments are therefore ablations at three levels:

  • Stage I (App. C.1): learned instance association raises instance accuracy from 69.47% to 82.93%, and from 35.30% to 49.59% when two objects share a category.
  • Stage II as a whole (Table 1): this table is an ablation, not a conventional baseline comparison. Stage I is kept fixed and our Stage II is replaced by four temporal models. ABR@8 falls from 59.60% to at most 38.90% (IID) and from 61.44% to at most 49.10% (target assets).
  • Inside Stage II (Table 2, App. F): R improves task binding (role accuracy up to 89.62% from 74.95% / 81.06%), P improves the timing of state changes (Transition F1 .140 → .305 and .178 → .458), and combining their fields adds 2.02 / 1.46 points of Whole@8 over P+R.

Stage II is also not a stack of existing modules. Its core is the revision rule in Sec. 4.3. At every step the model weighs three options: keep the previous belief, adopt a candidate corrected by the new measurements, or adopt a candidate predicted from the executed action. A learned reliability sets the mix, and the accepted belief becomes the prior for the next step. This is how a failed grasp can correct a wrong HELD belief even though nothing in the world changed. Typed memory and local attention only help form the candidate; App. G shows their gains are smaller and overlap. As an additional check, we removed this revision path for relation/articulation and retrained (three seeds). Transition F1 drops by .026 (IID) and .036 (target assets), with both 95% intervals excluding zero. Recovery is essentially unchanged, the same timing-versus-recovery pattern as in Q2.

Q2 (W2). Why does E+R recover more while CERB times transitions better?

The two metrics ask different questions. Recovery asks: within two seconds after a change, was the full state right at least once? Transition F1 asks: did the belief change when the world changed, and not at other times?

E follows what the cameras currently show, so it scores a hit slightly more often. But it is much worse at registering that a relation has changed (relation F1 .195, against .469 with P). So yes, using P costs 1–3 points of recovery, because a hit needs every field right at the same step. In return, timing improves more than 2×, and a planner must know when a grasp or placement actually happened. The full readout is a deliberate trade-off, not a claim to be best in every column. The conclusion does not depend on it: all three readouts are far above every alternative in Table 1 on ABR@8 (by at least 20 points on IID and 11 on assets).

Q3 (W3). Does a better belief give better task success?

Sec. 5.3 and App. I run this test: same planner, same skill executor, same 18 starting states; only the belief changes. CERB succeeds in 27/54 rollouts, against 22, 17, 13 and 7 for the four alternatives, and is best or tied in every start type (Table 3).

Across the five systems, all four metrics correlate with success in the right direction (Spearman |ρ| ≥ 0.60 on both test sets). Transition F1 and ABR@8 are the closest on IID (ρ = 1.00 and 0.90). With five systems we treat this as indicative. App. I also shows the mechanism: changing only the "container is open" probability flips the planner's choice between OPEN and GRASP.

Q4. Objects that appear after the episode starts

  • Named in the instruction but not visible at first (e.g., inside a closed refrigerator): handled. Stage I looks for every named object in every frame. Until the object is seen, its state stays uncertain; once seen, its belief is updated; when hidden again, the belief is kept. Seven of the 18 downstream starts begin with the target hidden.
  • Not named in the instruction: not tracked, by design. CERB maintains the state of the objects the task needs, not a map of the whole scene. Creating slots for new instances mid-episode is an extension we note in Sec. 6.

Q5 (W4). Unseen categories, new relations, dependence on the observer

  • New objects. Recognition belongs to Stage I, which grounds objects from the instruction text over frozen SigLIP 2 features, not from a fixed class list. On target objects whose assets were withheld from temporal training, the ABR@8 advantage holds (61.44% vs 49.10%).
  • New relations. Relations are an explicit list (HELD, IN, ON, NONE, UNKNOWN), which is what makes the belief checkable and usable by a planner. A new relation needs a new output state and labels; the revision rule itself does not change.
  • Observer errors. The observer is not an external module assumed to be accurate. It is our learned Stage I and it is imperfect, so every reported result already includes its errors. It also tells Stage II how much evidence supports each measurement, and Stage II keeps its previous belief when there is none.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Explain our paper in plain language

Since this paper has been officially withdrawn, we have redacted the detailed discussion of our core ideas to protect unpublished intellectual property.

Since this paper has been officially withdrawn, we have redacted the detailed discussion of our core ideas to protect unpublished intellectual property.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.