acceptodds
Under review as a conference paper at ICLR 2027

Analysing When a Learned Subset of Objects Explains a Robot Policy: Discovery, Removal, Consequence

Abstract

Object-centric robot policies promise to generalize across scenes and to reveal which objects a robot’s decisions rest on. We study In-Loop Subset Explanations (ILSE), policies that select a subset of objects inside the model, train the selec- tion end-to-end and present it as the explanation, and ask when this explanation is earned. We state three conditions for increasingly strong readings of the kept set: The selection must be discovered by the model rather than imposed (Discovery), excluded objects must be absent from the computation (Removal), and it must matter beyond a random subset of the same size (Consequence). We give a test for each condition and apply them to graph-based object-centric policies on three ma- nipulation benchmarks. Excluded objects still reach the action through standard operators, and correcting the removal lowers success by up to 24% relative on CALVIN, mostly through the reductions over excluded objects, and on one split turns an advantage over a pixel policy into a deficit. No tested design point satis- fies all three conditions: A labeled selector beats random selection at a matched count, but its choice is imposed by annotations, while a label-free selector that meets Removal and Discovery falls below random selection on average.

Primary area:
interpretability and explainable AI
Keywords:
Explainable robot learning, Object-centric policies, Object-graph policies, Sparsification, Faithfulness, Removal, Imitation learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yt3XOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

In robotics domains, it's quite common to assume that the world is composed of objects since that usually gives us good generalization. When the observation is object-factorized, the policy (or some other module) selects the objects of interest for the given state, and makes decisions based only on that set. Nevertheless, this assumption might be silently broken, or even though the method is designed to leverage object-factorization, it might not be doing so because (1) it might be overfit to selecting the same set of objects as in the training set, or (2) unselected objects' might be leaked through selection, or (3) it doesn't perform better than random object selection. This work proposes three tests to check each of these conditions to see if the "object-factorization helps" claim is true.

Critical strengths

The proposed tests could serve as a sanity check for methods that use the object-factorized formulation.

Critical weaknesses

It took my considerable effort to understand the paper. It is as if the paper is shared as an internal document with a colleague who works on the same topic and can understand what is meant by each phrase, rather than a manuscript with clear definitions intended for a general audience. I'm not saying this as a bad thing---it's just that the text is really hard to understand because of many unclear definitions and phrases. It could be because I'm not very familiar with the topic, but even then I could say that the motivation and narrative are not very clear from the abstract and introduction. Likewise the experiment settings and the results that accompany them.

The reasoning behind some of these tests is not well-motivated. Especially for the discovery condition, why exactly do we want to limit the method to train its selector from the task error only? Let's say there's indeed a labeled dataset that tells which of the objects should be selected. Why would this be a limitation, if the selector then can generalize?

Questions for authors
  • What methods are we considering? Behavior-cloning policies, reinforcement learning, and in which setting? MDP, POMDP?
  • Aren't selectors permutation-invariant by design? I can train a selector using the labels (which I'm not allowed to if I understand the discovery condition correctly) for each of the inputs without providing an order. Same for the cardinality. An object-level classifier could be trained on the annotation list which would not depend on the cardinality. In any case, these scenarios are not very clear.
3 comments

We thank the reviewer for the careful reading and the constructive comments. We revised the paper mainly for clarity:

  • Framing. We now present the work as the ILS Explanation framework. Section 3 first defines the setting formally (a selector gϕg_\phi that selects a subset StS_t of the objects, and an action generator fθf_\theta that acts on it) and then states each condition with a formal requirement, the ways it can fail, and its test.
  • Conditions. Discovery now requires that the selection uses no signal about which objects are relevant (annotations, motion labels or other heuristics). Removal is defined as invariance of the action to the excluded objects, given the selection, and we added three tests for information carried by the selection itself. Consequence compares against a random selection with the same number of objects and the same robot parts.
  • Presentation. New Figures 1 and 2 illustrate the conditions; Table 2 reports standard deviations, marks the best results and has no unexplained gaps. Related Work positions the paper against Azzolin et al. (ICLR 2025), and the appendix follows the order of the main text.

While revising, we found that the corrected removal path of the submission still leaked the positions of removed objects. Table 2 now reports the fully corrected path, which has no significant performance difference to the model with leaked positions. Therefore, the interpretation of the results is unchanged. If time permits, we would be grateful if you could read the revised Introduction and Section 3, where most changes are. We would really appreciate if the reviewer could now tell us if the paper is easier to follow and more understandable, as we want to make it available for anyone and not just our colleagues.

W1: Readability. Thank you for this direct feedback as it led to the largest changes. The Introduction now states the problem more directly and introduces the ILS Explanation framework explicitly. Section 3 defines the setting formally (a selector that picks a subset of the objects, and an action generator that acts on it) and then treats each condition with the same structure: A requirement, the ways it can fail, and its test. Figure 2 illustrates each condition, and Section 4 follows the same order.

W2: Why Discovery? D-C does not say that supervised selection is bad for performance or generalization. A labeled selector can select well and generalize (Section 3.4). It concerns only the explanation claim: Whether the selected subset shows what the model found. A selector trained on an annotation list learns to reproduce it (AUROC ≥ 0.998 on CALVIN), and in our experiments it performs about as well as a policy that is simply handed the annotated objects (Table 2). Its subset therefore shows what the annotator decided, not what the model relies on, and cannot reveal, for example, a bias in the data that the annotator did not know (Sections 3.1 and 4.2).

Q1: Setting. We consider policies trained by behavior cloning (imitation learning) and evaluated in closed loop. At each step, a set of objects is extracted from the benchmark's static cameras, so the policy observes the scene only partially through these objects. We do not consider reinforcement learning.

Q2: Permutation invariance and cardinality. Yes, the selectors are permutation-invariant over the input order. Our former "permutation test" did not permute the input. It reassigned the relevance labels to other objects to verify that the code of a label-free selector never reads them. We renamed it the label-shuffle test to avoid this confusion (Section 3.1, Appendix A.4). An object-level classifier trained on the annotation list is indeed independent of order and cardinality, but it still fails D-C, because it is trained on a relevance signal: its selection reproduces the annotation. The cardinality failure refers to something else: a budget taken from the annotation (e.g., the number of annotated objects per task) tells even a label-free selector how many objects to keep.

Thank you for your response.

Its subset therefore shows what the annotator decided, not what the model relies on, and cannot reveal, for example, a bias in the data that the annotator did not know (Sections 3.1 and 4.2).

Yes, but can't we just treat the annotator as part of the model? Correct me if I'm wrong. Let's say there's a selector model that is trained with labels. And it shows some generalization in the test set as well. Doesn't this show that the downstream model relies on it? Selector model is just another component of the big architecture. I'm not sure what I am missing here. I'm asking why do we want to specifically train the selector model with the downstream loss? You're claiming that it will memorize labels, but we don't know that---I don't think the test set performance will just drop to zero.

Thank you for the follow-up. We think the question separates two claims an explanation can make, and we agree with you on the first.

1. "The action depends only on the selected objects." If the annotator is treated as part of the model, a labeled selector with corrected removal supports this claim: its action depends only on the selected subset (Removal), and the subset is more informative than a random one (Consequence). Our labeled selector satisfies both conditions (Table 2), and it can be useful and generalize well (Section 3.4).

2. "These are the objects the model found relevant." This is the claim that object-selecting policies such as SIR make when they present the subset as revealing what a task requires, or a bias in the data. Discovery concerns only this claim. A selector trained on an annotation list learns to predict that list, and, as you say, it generalizes: it reaches an AUROC of at least 0.998 against the annotation on held-out frames. By "reproduces" we do not mean memorization or a drop in test performance. We mean that on new scenes it shows the objects the annotator would have marked. Its subset therefore reports a judgment that was already written down, not one the model formed. If the annotation misses an object the task needs, the labeled selector learns to drop it too, so the explanation cannot reveal the gap. A selector shaped only by the policy's loss can, because nothing tells it which objects to keep. Our label-free selectors, for example, keep different objects than the annotation (Section 4.2).

Why the downstream loss. Training the selector only through the policy's loss is not required for good performance or for a faithful explanation; it is required only if one wants to read the subset as what the model found. In our experiments, the labeled selector performs about as well as a policy that is simply given the annotated objects (Table 2), which illustrates that its selection adds little beyond the annotation itself.

Reviewer_hRGnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores In-Loop-Selection-Explanations (ILSE) as they pertain to policies that operate on object centric representations for robotic manipulation. Specifically, the focus is on policies that select a subset of objects from the scene and operate only on them to produce actions. So, for a task like "place the straw inside the glass," superfluous objects present in the scene, such as plates or apples, will be absent from the selection. To this end, the authors present an analytical formalization in three conditions: Discovery (object selection must be independent of ground truth annotations), Removal (action must be agnostic of excluded objects), and Consequence (selection must be better than random). Importantly, the authors discover a leak: even when an object is not selected, its information can still affect the action output. None of the tested approaches satisfy all 3 conditions simultaneously, as defined.

Critical strengths

This work addresses an important question in explainability for robot learning for object selection based policies: whether the choices made here genuinely explain the decisions made by the policy.

Critical weaknesses

Conceptual novelty of the analysis framework proposed is somewhat limited and more or less formalizes existing concerns about explanation faithfulness, information leakage, and meaningful feature selection.

Concerns about the informative scope of the conditions:

  • Discovery (D-C) tests dependence on object annotations, rather than whether the model discovers task-relevant objects. Penalizing supervised object selection seems unjustified, especially when Motion-Grounded Selection is a rough proxy for task-relevant object saliency, but passes D-C.
  • Removal (R-C) encodes independence from excluded objects conditional on the selected subset, meaning information can be transmitted through the selection itself and this is not examined. To illustrate this, consider the task “place the apple inside the drawer.” Here, the selector may observe the drawer state (object feature, x_i, as per Sec. 3) and if the drawer is open, it selects the handle (prompting the policy to open it) and the drawer is closed, it selects the apple (prompting the policy to pick it up). Effectively, the policy doesn’t observe the drawer (it was excluded), but the selection of the apple communicates that it was open.

The central result (no tested approach satisfies all three conditions) relies on a single label-free selection approach (satisfy D-C and R-C, but fail C-C by performing worse than random selection). On CALVIN, retaining the links does make the approach beat random (within margin of error) which means C-C does not fail (this is acknowledged in the paper, but weakens the primary conclusion).

In the Tab. 2, - indicates “not run or not applicable.” If the reason for not running it is something other than it not being applicable, this should be explicitly stated/highlighted. Label-free has no results on CALVIN ABC->D or RoboCasa, VGIB has no RoboTwin results, Labeled (corrected)’s RoboTwin distractor evaluation is marked “not run,” etc. None of these have any clear justifications.

Questions for authors

Questions:

  • Could the authors provide additional analysis or experiments addressing the limitations of D-C and R-C discussed above?
  • Can missing entries in Table 2 either be populated or justified?
  • Why are the random and learned selectors trained with different masking strategies on RoboTwin, unlike on CALVIN?
  • Why were the 8 pick-and-place tasks (33% of the benchmark) in RoboCasa dataset excluded from the main analysis in the paper?
4 comments

We thank the reviewer for the careful reading and the constructive comments. We revised the paper mainly for clarity:

  • Framing. We now present the work as the ILS Explanation framework. Section 3 first defines the setting formally (a selector gϕg_\phi that selects a subset StS_t of the objects, and an action generator fθf_\theta that acts on it) and then states each condition with a formal requirement, the ways it can fail, and its test.
  • Conditions. Discovery now requires that the selection uses no signal about which objects are relevant (annotations, motion labels or other heuristics). Removal is defined as invariance of the action to the excluded objects, given the selection, and we added three tests for information carried by the selection itself. Consequence compares against a random selection with the same number of objects and the same robot parts.
  • Presentation. New Figures 1 and 2 illustrate the conditions; Table 2 reports standard deviations, marks the best results and has no unexplained gaps. Related Work positions the paper against Azzolin et al. (ICLR 2025), and the appendix follows the order of the main text.

While revising, we found that the corrected removal path of the submission still leaked the positions of removed objects. Table 2 now reports the fully corrected path, which has no significant performance difference to the model with leaked positions. Therefore, the interpretation of the results is unchanged. If time permits, we would be grateful if you could read the revised Introduction and Section 3, where most changes are and give us feedback if the paper is now easier to follow and more understandble.

W1: Novelty. The concerns themselves (faithfulness, leakage, informative selection) are known and we agree on this part with the reviewer. Our contribution is a framework that states them as three testable conditions for policies that present an in-loop selection as their explanation, and the tests themselves. Compared with the closest work (Azzolin et al., ICLR 2025), we add two leak routes (the number of excluded objects and their slot positions), an exact construction test that also applies beyond graph networks, the closed-loop setting where the cost of each leak can be measured, and the Discovery and Consequence conditions (Section 2, Section 3.2).

W2: Discovery and supervised selection. We agree that supervised selection can be useful, and the paper does not penalize it as a design. D-C only decides whether the selected subset can be read as what the model found: a selector trained on annotations reproduces them (AUROC ≥ 0.998 on CALVIN), and it performs about as well as a policy that is simply handed the annotated objects (Table 2), so its subset shows the annotation. Following your point, D-C now excludes any relevance signal, so Motion-Grounded Selection, whose labels define relevant objects by motion, fails D-C. It also does not beat random selection at its own count (+0.003 [−0.099, +0.105] on CALVIN).

W3: Information through the selection (drawer example). This is correct, and R-C does hold the selection fixed by definition. The revised Section 3.2 names this case ("Failure through the selection itself", with an analogous example), and Section 4.3 and Appendix A.5 report three new tests on CALVIN:

  1. When the excluded objects change, the label-free selection changes in nearly every frame and the labeled one rarely (3% of frames on D→D).
  2. No selector's choice reveals the state of an excluded drawer, slider, LED or light bulb beyond a random choice (largest excess +0.046), while a selector we built to encode the drawer state is detected perfectly.
  3. A policy trained on random selections gains nothing from the labeled selections (+0.03 [−0.03, +0.09]), so the labeled advantage over random appears only together with the policy trained with it.

W4: The central result rests on one label-free selector. We now test two label-free selectors with different gradient estimators (sampled masks and relaxed top-k) plus VGIB. With the robot's own parts kept, the CALVIN difference to random selection is +0.06 [−0.12, +0.23]: The interval contains zero, so C-C is not detected, not met. We now state this decision rule explicitly (Section 4). The result is the same on every benchmark and with a larger budget that is not derived from the annotation (Appendix A.6).

W5 / Q2: Missing entries in Table 2. Almost all entries of the submitted table are now filled: Label-free on ABC→D and RoboCasa, VGIB and Leaky reductions on RoboTwin, and Corrected under distractors. "--" now only marks cases where a condition does not apply (C-C requires a design that satisfies R-C). The label-free models are still only available on CALVIN D->D but should be finished by the end of the week and updated in the paper. All new results do not change the conclusion from the original submission.

Q3: Masking of random selections on RoboTwin. Appendix A.1 now states, per benchmark, how each random selection is trained: One random subset per scene, or a new subset at every training step. On CALVIN both kinds occur as well, which the submission described incorrectly. On RoboTwin, the label-free result without robot parts depends on this choice (−4.3 [−11.9, +3.2] against the first, +3.6 [+1.1, +6.1] against the second), and we report both (Appendix A.6). The comparison that C-C uses, with the robot's parts kept on both sides, shows no difference either way.

Q4: Pick-and-place on RoboCasa. On the 8 pick-and-place tasks every object-graph policy stays at 0.10–0.12% success, so these tasks cannot separate any variants. We report all 24 tasks in Appendix A.3.

Reviewer_hRGnOpenReview ↗

Reviewer Reponse

Thank you for your response. Most of my concerns are addressed. However, I'm not convinced about this work's novelty and find it to be somewhat incremental. I will keep my score.

Thank you for your response. We are glad that most of your concerns are addressed.

Regarding novelty: "incremental" suggests that there is prior work our contribution builds on closely. The only such work raised in the reviews is Azzolin et al. (ICLR 2025), which overlaps with one part of one of our three conditions, as we discuss in Sections 2 and 3.2. If you have other works in mind that already state when a policy's selected subset can be read as its explanation, or that test these conditions in closed loop, we would be grateful for the references. They would help us position the paper more precisely in the final version.

Reviewer_Ufz6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates object-subset explanations in robot policies using three conditions named 'Discovery, Removal, and Consequence'. The main contribution is very empirical. The paper mainly applies an explicit constructed-input test to assess whether excluded objects still influence the policy, and measures how correcting this information leakage affects closed-loop performance on several robotics benchmarks.

Critical strengths
  1. The finding that excluded objects still affect computation is interesting.
  2. The empirical work is comparatively solid.
Critical weaknesses
  1. The paper lacks novelty. Concepts like hard selection, restricted aggregation, and difficulties in training faithful explanations has already been discussed in [1]. This paper mainly takes the concept and apply on robotics benchmark. The conceptual contribution is limited.
  2. The presentation of the paper is very hard to follow. Discovery, removal, and consequence are three different problems, I can see that there is a workflow connecting these problems, but the paper constantly bundle them together without explaining the problems clearly.
  3. The paper lacks visual examples. The only two figures are not very informative. The numbers in the tables are not bolded. Also, uncertainty estimates appear only in the appendix. Including them in the main results would help with the reliability of the comparisons.
  4. There is only very limited mathematical formulation. I think the paper can use formulations and mathematical language to make their points better and to make the paper easier to follow.

[1] Steve Azzolin, Antonio Longa, Stefano Teso, and Andrea Passerini. Reconsidering Faithfulness in Regular, Self-Explainable and Domain Invariant GNNs. ICLR, 2025.

Questions for authors
  1. What does the “no-choice” diagnostic actually measure? For k = 1, the score vectors (0.999, 0.1) and (0.1, 0.999) both satisfy the stated no-choice criterion but select different objects.
  2. Are anonymous supplementary artifacts available? Please provide executable leakage/correction test and any visualizations of the tasks.
  3. Have the authors investigated tasks on real robots? Do these results still hold with perception error and object-tracking instabilities?
3 comments

We thank the reviewer for the careful reading and the constructive comments. We revised the paper mainly for clarity:

  • Framing. We now present the work as the ILS Explanation framework. Section 3 first defines the setting formally (a selector gϕg_\phi that selects a subset StS_t of the objects, and an action generator fθf_\theta that acts on it) and then states each condition with a formal requirement, the ways it can fail, and its test.
  • Conditions. Discovery now requires that the selection uses no signal about which objects are relevant (annotations, motion labels or other heuristics). Removal is defined as invariance of the action to the excluded objects, given the selection, and we added three tests for information carried by the selection itself. Consequence compares against a random selection with the same number of objects and the same robot parts.
  • Presentation. New Figures 1 and 2 illustrate the conditions; Table 2 reports standard deviations, marks the best results and has no unexplained gaps. Related Work positions the paper against Azzolin et al. (ICLR 2025), and the appendix follows the order of the main text.

While revising, we found that the corrected removal path of the submission still leaked the positions of removed objects. Table 2 now reports the fully corrected path, which has no significant performance difference to the model with leaked positions. Therefore, the interpretation of the results is unchanged. If time permits, we would be grateful if you could read the revised Introduction and Section 3 and tell us if this makes the paper easier to follow and more understandable

W1: Novelty against Azzolin et al. (2025). Thank you for the pointer. We now discuss this work in Section 2 and Section 3.2. Our requirements R1–R3 coincide with their design rules for self-explainable graph networks, and the difficulty of training a hard selection is our R4. Beyond them, the paper contributes:

  1. Two leak routes their rules leave open, the number of excluded objects and their slot positions, with requirements that close them.
  2. An exact construction test that detects all routes and applies beyond graph networks (e.g., DynamicViT, Appendix A.5).
  3. The closed-loop setting, where removal must happen at training time and closing the leaks costs up to a quarter of the success.
  4. The Discovery and Consequence conditions, which their analysis does not cover.

W2: Presentation. We restructured Section 3. It starts with the setting (selector and action generator), then treats each condition in its own subsection with the same structure: a requirement, the ways it can fail, and its test. Section 3.4 then explains how the conditions relate and why all three are needed. Section 4 follows the same order.

W3: Figures, bolding and uncertainty. Figure 1 now shows the pipeline and the three questions the conditions answer. The new Figure 2 shows, for each condition, what it requires and how it fails. Table 2 reports the standard deviation over seeds for every cell and marks the best result per column in bold.

W4: Mathematical formulation. Section 3 now defines the policy as at=π(Ot,c)=fθ(gϕ(Ot,c),c)a_t = \pi(\mathcal{O}_t, c) = f_\theta(g_\phi(\mathcal{O}_t, c), c) and states each condition formally:

  • R-C as π(Ot′,c)=π(Ot,c)\pi(\mathcal{O}'_t, c) = \pi(\mathcal{O}_t, c) for every scene Ot′\mathcal{O}'_t with the same selected objects but different excluded ones;
  • C-C as J(gϕ)>J(g~)J(g_\phi) > J(\tilde{g}) for a random selector g~\tilde{g} of the same size;
  • the non-decision diagnostic through the normalized gap at the cut (Eq. 1).

Q1: The no-choice diagnostic. You are right: the old criterion did not measure a decision. We replaced it (Section 3.1, Appendix A.4) by the normalized gap at the cut, δt=(s(k)−s(k+1))/(s(1)−s(nt))\delta_t = (s_{(k)} - s_{(k+1)}) / (s_{(1)} - s_{(n_t)}).

  • Your examples (0.999, 0.1) and (0.1, 0.999) with k=1k = 1 both give δt=1\delta_t = 1, a clear decision.
  • (0.999, 0.998, 0.1) gives δt≈0.001\delta_t \approx 0.001, a cut decided by numerical noise.

We also report when top-kk must fill slots with low-scored objects or cut among nearly equal high scores. The labeled selectors decide clearly (median gap above 0.8); the label-free selector cuts with a small gap (median 0.05–0.07).

Q2: Anonymous artifacts. We will post a anonymous link by the start of next week, visible to reviewers and ACs. It contains the construction test (leakage and correction), the label-shuffle test, the evaluation records, and rollout videos of each task.

Q3: Real robots. We have not tested on real robots, and the paper states this as a limitation. Removal is a property of the computation after the selection, so the construction test holds independently of perception quality. Discovery and Consequence, however, depend on the objects the policy receives, but from early experiments with fine-tuned YOLO networks the object detection quality was almost as good as ground-truth from the simulation.

Thank you for your response.

  1. Although several of my original concerns were addressed, my main concern about novelty remains. The central architecture in this paper overlap with Azzolin et al. Although the additional leakage routes and robotics experiments are useful I still find the overall contribution incremental.

  2. The newly found positional leakage also raises a concern about the construction test. If the original corrected implementation leaked information, why did the original test not detect it? Please clarify where the leakage occurred and how the revised test now checks the full policy computation. Also, finding no significant performance difference does not necessarily establish that the two implementations are the same. Could you provide the paired differences to support the claim that the performance conclusions remain unchanged?

Overall I appreciate the authors' efforts to address the comment, and the work's empirical contribution. However my concern about novelty and the validation of the construction remain and so I will maintain my score of 2(weak rejection).

We thank the reviewer for the follow-up and for acknowledging the empirical contribution.

Novelty: the overlap with Azzolin et al. is limited to R1–R3. The paper's contribution is the ILS Explanation framework: three conditions that are each necessary before a selected subset can be read as a policy's explanation. The overlap concerns only the design rules R1–R3, which are one part of one of these three conditions:

Azzolin et al. (2025)Ours
Scopegraph neural networks (regular, self-explainable, domain-invariant)any policy that selects a subset of its input objects; not bound to graphs (we also test token pruning in a vision transformer, DynamicViT)
Discovery (the model found the subset itself)—new condition and test
Removal: design ruleshard scores, readout over the explanation onlyR1–R3 coincide (stated in Sections 2 and 3.2)
Removal: leak routesvalues of excluded nodes reaching the readoutadds the count of excluded objects and their slot positions
Removal: verificationfaithfulness metrics, which they show to disagreeexact construction test, plus three tests for information carried by the selection itself
Consequence (better than a random subset of the same size and robot parts)—new condition and test
Settingstatic predictionclosed-loop control, with the cost of each leak measured by retraining

These additions change the verdicts: the labeled selector satisfies Removal but not Discovery, and the label-free selectors satisfy Discovery and Removal but not Consequence. Neither result follows from the design rules alone.

Where the leak occurred and why the original test missed it. In the submitted corrected path, removed objects were zeroed but kept their token slots. The action generator adds a positional embedding per slot, so the pattern of empty slots revealed which objects had been removed. The original construction test compared the two inputs at the encoder output, before this embedding is applied, and therefore passed. The revised test compares the action, the output of the full policy, in float64 with identical denoiser noise, and it includes the unpacked path as a configuration built to fail. The results are below; we will add the CALVIN rows to Table 10 (Appendix A.5) in the next revision.

packed pathunpacked path (built to fail)
CALVIN D→Dpasses on 4/4 seeds, max. difference 8.5×10⁻¹⁴fails on 16/16 batches, min. 6.0×10⁻³
CALVIN ABC→Dpasses on 4/4 seeds, max. difference 5.5×10⁻¹⁴fails on 16/16 batches, min. 2.3×10⁻²
RoboCasabit-identicalfails, 2.0×10⁻²

On RoboTwin the corrected path was packed from the start, so its results are unaffected. This bug also illustrates our point: a slot leak is easy to miss even when the design rules are followed, which is why the framework includes an exact test of the full policy. Paired differences. We agree that a non-significant difference does not show equality. The paired differences between the packed and the unpacked path, over seeds, are:

packed − unpacked95% intervalrelative
CALVIN D→D+0.036[−0.151, +0.223]+2.2% [−9.3%, +13.8%]
CALVIN ABC→D−0.071[−0.234, +0.093]−4.1% [−13.7%, +5.4%]
RoboCasa−0.34 points[−1.68, +1.01]—

With four seeds, these intervals cannot rule out differences of about ±10%. Our conclusions, however, do not rely on the two paths being equal, because we recomputed every contrast they rest on with the packed path:

CALVIN D→DCALVIN ABC→DRoboCasa
Cost of closing the leaks (packed / unpacked)22% / 24%25% / 22%15% / 14%
Share of the cost recovered by reopening only the reductions (packed / unpacked)87% / 88%84% / 81%101% / 101%

The condition verdicts of all variants in Table 2 are also unchanged. A few diagnostic contrasts are still computed on the unpacked path; Table 7 marks them.

Both versions are reported in the appendix (Table 6), and we corrected the submission's claim that the unpacked path satisfies R-C.

Comments on the paper

1 comment
Area_Chair_12XDOpenReview ↗

Do the rebuttals address your concerns?

Hi Reviewers,

If you haven't done already, please read the author rebuttals and to let them know if it addresses the concerns raised in the original reviews.

Thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.