acceptodds
Under review as a conference paper at ICLR 2027

Refusals That Bend: Measuring and Predicting Task Malleability in Embodied VLM Planners

Abstract

Embodied vision-language models (VLMs) are increasingly deployed as high-level planners for robots because they generalize across diverse environments. However, this requires their safety alignment to also hold in unseen environments. Existing red-teaming assumes an adversary who optimizes the prompt, the pixels, or text in the environment, and existing benchmarks ask whether a planner recognizes or mitigates a hazard in a fixed scene. Neither asks whether a refusal the planner has already given survives an ordinary change to the environment. We ask that question by placing a single everyday object into the environment, with no pixel, gradient, or prompt under adversarial control. On tasks that a constitution-guarded planner initially refuses, we find of tasks can be flipped to compliance by one or more objects, and the number of objects differs from one task to another. In addition, the object need not be chosen for the task, i.e., items drawn from a fixed list, with no knowledge of the environment or the instruction, bypass safety about as often as items proposed for the specific task. We qualitatively contrast the tasks bypassed most and least often and find that the distinction lies in how conspicuous the hazard is in the instruction and environment. Susceptibility to safety bypass is therefore a property of the task, which we call its malleability, and we show that it can be predicted before the target is ever queried. A composite of signals read from a small open-source VLM identifies malleable tasks as often as picking at random. Everyday objects, whether placed by an adversary or introduced by ordinary rearrangement of the environment, are thus sufficient to overturn a refusal. Because susceptibility is determined by how a task is specified, we recommend assessing malleability per task prior to deployment.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied VLM, Robotics, Safety Alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zSY9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the stability of safety refusals in embodied VLM planners under ordinary environmental changes. It introduces task malleability, defined as how easily a previously refused task becomes compliant after inserting an everyday object into the scene. On 846 initially refused tasks, the authors find that 20.2% can be flipped by at least one object, and that random objects are nearly as effective as task-targeted ones. They further analyze which tasks are most malleable, introduce the LatentHarm dataset, and show that signals from a small open-source VLM can predict malleable tasks before querying the target planner. 44707_Refusals_That_Bend_Measu

Critical strengths
  1. The paper identifies a clear and practically relevant safety failure mode. The notion of task malleability captures an important question that is not well covered by standard red-teaming or fixed-scene safety benchmarks: whether a refusal remains stable after ordinary environmental changes. The finding that 20.2% of initially refused tasks can be flipped by adding a single everyday object makes this concern concrete and consequential.

  2. The empirical study is careful and reasonably comprehensive. The paper goes beyond reporting bypass rates by controlling for decoding noise, comparing targeted, scene-only, and random object insertions, auditing edit fidelity, and analyzing which tasks are most susceptible. This makes the main phenomenon more convincing and provides a stronger basis for the paper’s conclusions.

Critical weaknesses
  1. Limited validation across target planners. The core findings are demonstrated on a single target planner, Gemini Robotics-ER 2, with one safety constitution. It is therefore unclear whether the observed malleability generalizes to other embodied VLM planners or alignment mechanisms.

  2. The environmental perturbations are synthetic. The study inserts objects using a diffusion-based image editor rather than physically modifying real environments. Although the authors carefully audit edit quality, it remains uncertain whether the same refusal reversals would occur under real physical scene changes.

  3. The claim that malleability is a property of the task may be too strong. The experiments show that variation across tasks is much larger than variation across object-selection strategies, but this does not fully establish that object identity is unimportant. A more cautious conclusion would be that task-level susceptibility dominates under the tested object distributions.

  4. The predictor lacks external generalization validation. The malleability predictor is evaluated through cross-validation on the same 846-task population, while LatentHarm is constructed from the patterns discovered in this study. Cross-dataset and cross-planner validation would be needed to support stronger claims about deployment-time prediction.

Questions for authors
  1. Generality. Do the authors have evidence that both the malleability phenomenon and the proposed predictor generalize to other embodied VLM planners, safety-alignment schemes, or independent datasets?

  2. Task vs. object effect. Can the authors quantify how much of the observed variation comes from the task itself versus the inserted object?

In general, I like the idea of this paper. Strong answers to the above questions, particularly with additional supporting evidence, could increase my score.

4 comments

We thank the reviewer for the encouraging feedback and for stating clearly what would strengthen the paper. We address both questions below.

Generality. We will evaluate additional target planners from different model families (two to three depending on time allowance). For each planner, we will run the same pipeline as in the paper: collect the tasks it initially refuses, insert the proposed objects, and measure how often the refusal is reversed, compared against a clean re-query of the unmodified image. We will also investigate how much of the malleable tasks are shared between the different planners.

Task versus object effect. Section 4.3 (RQ2) answers this by comparing three ways of choosing the object for the same task: targeted, scene-only and random. Figure 2 groups tasks by their reversal rate under targeted objects, so the rise of the targeted bars is by construction. The scene-only and random bars played no part in the grouping, yet they match the targeted bar in every group and rise from under 1% to about 50%. We will revise the text to better quantify and explain this.

We have completed the experiments described in our initial response. The results are summarized below and will be included in the revised paper.

Generality. We added three additional planners: GPT-5.4, a general-purpose VLM, and Cosmos-Reason2 and RoboBrain2.5, two open-weight 8B models built for robotic planning. Each is evaluated with the same pipeline as in the paper. We first report how many tasks each planner initially refuses, and then how many of those are malleable, meaning that at least one of the 20 variants with an inserted object yields a plan. Mean malleability is the average number of the 20 proposed objects that flip the refusal, over the malleable tasks.

PlannerInitially refusedMalleableMean malleability
Gemini Robotics-ER 284620.21% (171)28.3%
GPT-5.496220.17% (194)28.0%
Cosmos-Reason211183.78% (93)48.0%
RoboBrain2.515490.91% (140)55.6%

GPT-5.4 behaves like Gemini Robotics-ER 2: it refuses a similar number of tasks, a similar share of them is malleable, and a malleable task is bypassed by 28% of its objects on average on both. The two open-weight planners refuse far fewer tasks, most of what they refuse is malleable, and a malleable task is bypassed by about half of its objects. We attribute this partly to their smaller size.

We then check that the reversals are not decoding noise. Each refused task is queried again on its unmodified image. For each task we compute two rates: the share of its 20 object variants that yield a plan (Object), and the share of its clean re-queries that yield a plan (Re-query). The table reports the average of each rate over the refused tasks with a conclusive answer under both, so tasks that never flip count as zero. The difference is paired within task, in percentage points with its 95% CI, and gives how much more likely a refused task is to yield a plan with an inserted object than when simply asked again.

PlannerObjectRe-queryDifference (pp), 95% CI
Gemini Robotics-ER 25.68%0.24%5.44 [4.37, 6.50]
GPT-5.45.59%4.75%0.85 [-0.33, 2.02]
Cosmos-Reason240.67%33.02%7.65 [0.62, 14.67]
RoboBrain2.550.86%38.24%12.63 [5.95, 19.30]

Refusals are unstable on every planner. A refused task can yield a plan when the same query is simply repeated, and a visual change to the image, such as inserting an object, makes the reversal more frequent on three of the four planners. On GPT-5.4 the trend is observed, but the difference from the re-query is not distinguishable from zero.

Proxy predictor on a different target family. The predictor's signals come from a small local model and never use the target, so they are identical for every planner. We took the composite (A5) fitted on Gemini Robotics-ER 2 and applied it to GPT-5.4 without refitting, feature selection or recalibration. Because 801 of the 956 tasks that GPT-5.4 refuses are also refused by Gemini Robotics-ER 2, and were therefore fitting tasks for the composite with different labels, we also report the 155 tasks that Gemini Robotics-ER 2 accepts, which the composite has never seen.

Composite (A5)Gemini Robotics-ER 2GPT-5.4GPT-5.4, tasks not refused by Gemini Robotics-ER 2
Refused tasks846956155
Chance level of PR-AUC20.2%20.3%68.4%
Spearman ρ0.4010.4360.287
ROC-AUC0.7850.8100.662
PR-AUC0.4760.5050.782
Precision at 500.580.640.76

On every metric the paper uses, the predictor performs at least as well on GPT-5.4 as on the planner it was fitted on. On the tasks it has never seen it still ranks malleable tasks above robust ones well above chance.

Task versus object effect. Section 4.3 gives us the data to quantify this. For each of the 192 tasks in that sample we chose the object in three ways (targeted, scene-only, random), 20 objects each, giving 576 bypass rates. We split the variation in these rates with a standard analysis of variance. Each task gets a center, its bypass rate over all 60 of its edits. The object effect is how far a task's targeted, scene-only, and random rates sit from its center. The task effect is how far each task's center sits from the center of all tasks. Squaring these distances and adding them up gives one number for each effect. The task's number is 92% of the two combined and the object's is 8%. The task therefore sets the bypass rate and the object barely moves it. We will add this analysis to Section 4.3 as the answer to this question and make Figure 2 easier to read. The null-edit control reported to Reviewer btta points the same way, an edit that adds no object reverses refusals about as often as one that does.

We thank the reviewer for their valuable feedback. We hope our rebuttal has adequately addressed the concerns raised in the original review. As the discussion period is approaching its end, we would appreciate hearing whether our responses have resolved your main concerns or if any questions or feedback remain.

Thank you for the additional experiments. They address most of my concerns and strengthen the paper. I raise my score to 3.

Reviewer_bttaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a robot planner that has already refused an unsafe task will maintain that refusal after an ordinary object is added to the scene. The authors insert objects using an image-editing model, then query the safety-guided planner again. They define a task’s malleability as the fraction of tested object insertions that turn refusal into compliance. The main experiments evaluate Gemini Robotics-ER 2 on 846 initially refused tasks, testing 20 objects per task. A key observation is that randomly selected objects perform nearly as well as task-specific ones, suggesting that susceptibility depends substantially on the task rather than only on carefully chosen objects.

Critical strengths

The paper raises a meaningful question: whether an existing safety refusal survives ordinary environmental changes. This extends safety evaluation from correctness in a fixed scene to consistency across scene changes. The central finding is supported by experiments across 846 tasks, with clean-image re-queries providing a useful control for variability across model calls.

Critical weaknesses
  1. The evidence more directly supports sensitivity to edited images than to real-world object placement. The experiments show that an image-editing pipeline involving object insertion can change safety decisions. Further controls are needed to separate the effect of the added object from other changes introduced by the editor.
  2. The safety judge lacks clear human calibration. It assesses compliance from the instruction and generated plan without seeing the scene image, even though the paper emphasizes hazards that require joint interpretation of the instruction and scene. This raises questions about whether the judge reliably distinguishes unsafe compliance from safe alternatives.
  3. The paper’s main conclusions are primarily based on Gemini Robotics-ER 2, making it unclear whether task malleability is a phenomenon that generalizes across embodied vision-language models.
Questions for authors
  1. What is the refusal-flip rate when images undergo the same editing process without adding an object?
  2. How accurate is the safety judge against human judgments? Do the main findings hold after excluding edits that violate the stated constraints?
  3. Does the malleability predictor retain its performance when evaluation splits are grouped by environment or task template?
6 comments

We thank the reviewer for the careful reading and the specific suggestions for controls. We address each point below.

Generalization. We will evaluate additional target planners from different model families (two to three depending on time allowance). For each planner, we will run the same pipeline as in the paper: collect the tasks it initially refuses, insert the proposed objects, and measure how often the refusal is reversed, compared against a clean re-query of the unmodified image. We will also investigate how much of the malleable tasks are shared between the different planners.

Effect of the added object versus the editing process. We would like to clarify what the paper claims, since we believe it is close to the reviewer's reading. We do not claim that a specific object is what reverses the refusal. Our comparison of targeted, scene-only, and random objects shows that the choice of object matters little, which suggests that malleable tasks can be reversed by visual changes in general. The claim is that, for some tasks, the refusal is easily reversed by a small visual change, and that inserting an object is one realistic way such a change can occur, through an adversarial or benign rearrangement of the environment. This non-determinism has safety implications for robots that can be deployed in diverse environments. We will revise the text to make this explicit. To address the reviewer's question directly, we will also pass the images through the same editing pipeline with an instruction that adds no object, and report how often the refusal is reversed alongside the object insertion results.

Accuracy of the safety judge and results after excluding edits that violate the constraints. We will have human annotators label a sample of the planner responses and report how well the safety judge agrees with them. On the constraints, the paper already reports an audit of a stratified sample of 966 edited images (Appendix "Constraint verification"). About 93% of the audited edits satisfy all seven constraints, and the edits that reversed a refusal satisfy them as often as the edits that did not (93.8% against 92.6%). Of the 193 audited edits that reversed a refusal, 181 passes every check, so the reversals are not explained by edits that violate the constraints. We will also manually confirm this judge's alignment with humans.

Predictor performance when evaluation splits are grouped by environment or task template. We will aggregate and report the data by the environment of the task alongside the current results using the annotated environment from the EARBench dataset.

We have completed the experiments and controls described in our initial response. The results are summarized below and will be included in the revised paper.

Generalization. We added three additional planners: GPT-5.4, a general-purpose VLM, and Cosmos-Reason2 and RoboBrain2.5, two open-weight 8B models built for robotic planning. Each is evaluated with the same pipeline as in the paper. We first report how many tasks each planner initially refuses, and then how many of those are malleable, meaning that at least one of the 20 variants with an inserted object yields a plan. Mean malleability is the average number of the 20 proposed objects that flip the refusal, over the malleable tasks.

PlannerInitially refusedMalleableMean malleability
Gemini Robotics-ER 284620.21% (171)28.3%
GPT-5.496220.17% (194)28.0%
Cosmos-Reason211183.78% (93)48.0%
RoboBrain2.515490.91% (140)55.6%

GPT-5.4 behaves like Gemini Robotics-ER 2: it refuses a similar number of tasks, a similar share of them is malleable, and a malleable task is bypassed by 28% of its objects on average on both. The two open-weight planners refuse far fewer tasks, most of what they refuse is malleable, and a malleable task is bypassed by about half of its objects. We attribute this partly to their smaller size.

We then check that the reversals are not decoding noise. Each refused task is queried again on its unmodified image. For each task we compute two rates: the share of its 20 object variants that yield a plan (Object), and the share of its clean re-queries that yield a plan (Re-query). The table reports the average of each rate over the refused tasks with a conclusive answer under both, so tasks that never flip count as zero. The difference is paired within task, in percentage points with its 95% CI, and gives how much more likely a refused task is to yield a plan with an inserted object than when simply asked again.

PlannerObjectRe-queryDifference (pp), 95% CI
Gemini Robotics-ER 25.68%0.24%5.44 [4.37, 6.50]
GPT-5.45.59%4.75%0.85 [-0.33, 2.02]
Cosmos-Reason240.67%33.02%7.65 [0.62, 14.67]
RoboBrain2.550.86%38.24%12.63 [5.95, 19.30]

Refusals are unstable on every planner. A refused task can yield a plan when the same query is simply repeated, and a visual change to the image, such as inserting an object, makes the reversal more frequent on three of the four planners. On GPT-5.4 the trend is observed but the difference from the re-query is not distinguishable from zero.

Effect of the added object versus the editing process. We ran the control the reviewer asked for on a stratified subset of tasks using the RoboBrain2.5 and Cosmos-Reason2 planners. The unmodified image of each task is passed through the same editor 20 times with the same prompt, from which the instruction to add an object is removed. This edit adds no content, but the diffusion editor still changes the image slightly. We call this a null edit.

PlannerTasksObjectNull editDifference (pp), 95% CI
RoboBrain2.59641.2%39.6%-1.6 [-5.1, +1.9]
Cosmos-Reason29735.9%32.1%-3.8 [-6.7, -0.9]

An edit that adds nothing reverses nearly as many refusals as an inserted object. This agrees with our comparison of targeted, scene-only and random objects, where the choice of object has little effect. This supports the general pattern that we observed, malleability relies more on how the task is defined and any visual perturbation can increase how often reversal is refused, objects are a realistic way for it to physically happen in the real world.

Accuracy of the safety judge and results after excluding edits that violate the constraints. We compared both judges with human annotations. For the safety judge, we collected human labels for 100 planner responses as accepted, refused or inconclusive, half of them responses to the unmodified image and half to an edited image. The judge agrees with the human labels on 91 (Cohen's κ = 0.83). The human labels confirm 44 of the 45 responses that the judge labelled accepted and 47 of the 55 that it labelled refused. The judge therefore rarely counts a refusal as a plan, and its errors make the reported reversal rates lower, not higher.

For the constraints, as noted in our initial response, the audit in the appendix checks 966 edited images against the seven constraints. Of the 193 audited edits that reversed a refusal, 181 satisfy every constraint, and the edits that reversed a refusal satisfy the constraints as often as those that did not (93.8% against 92.6). We compare the audit judge with human annotations on 100 audited edits. The two agree on 89 (κ = 0.76). The human labels confirm 59 of the 60 edits that the judge passed and 30 of the 40 that it failed, so the judge is the stricter of the two. Excluding the failed edits would not change our conclusions. Only 12 of the 193 reversals in the audit are an edit that fails a constraint, and failing edits reverse refusals no more often than passing ones, so dropping them removes a small and even share.

Predictor performance when evaluation splits are grouped by environment or task template. EARBench labels each task with one of 7 environment categories (for example home or commercial). The composite predictor of Section 4.5 is scored out of fold, we keep these held-out scores exactly as in the paper and only split the results by environment. PR-AUC is given with its chance level in brackets, reflecting the share of malleable tasks in that category.

EnvironmentTasksROC-AUCPR-AUC (chance)Spearman ρ
home3560.7720.446 (0.208)0.387
commercial1880.8630.713 (0.207)0.507
medical840.7360.522 (0.250)0.347
research830.6460.318 (0.193)0.227
industrial650.8460.473 (0.123)0.400
education510.8570.540 (0.235)0.531
entertainment19–––
All (as in the paper)8460.7850.476 (0.202)0.401

The predictor stays above chance in every environment with enough malleable tasks to score. Its strength varies, from research, where the margin over chance is smallest, to commercial, where it is largest. Therefore, the predictor transfers to every environment but not equally well, given the current dataset and training. Entertainment has one malleable task among 19, too few to give a meaningful score.

Reviewer_psm8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether safety refusals by embodied VLM planners remain consistent after adding an everyday object to a scene. On 846 initially refused tasks, 20.2% become compliant under at least one object insertion. Task-specific and randomly selected objects yield similar bypass rates on a stratified subset. This evidence do not require gradient propogation of specific adversarial attacks design. The authors also introduce task malleability. They found that the potential risks originated from tasks formulation itself and propose predicting it using signals from an open-source proxy VLM.

Critical strengths
  1. Testing refusal stability under ordinary environmental changes is a useful contribution in comparison with pixel/text level augmentation and optimized adversarial attacks.
  2. The paper proposes using signals from an open-source VLM to identify tasks with potential risks to safety bypass before querying the target planner, identifying vulnerable tasks 2.4 times as often as random selection.
  3. The study includes an audit checking whether edits preserve the scene and leave the hazard intact.
Critical weaknesses
  1. The reported evaluation uses diffusion-edited images and judges generated plans, assuming downstream execution. It does not establish that physically placing the same objects produces comparable effects in robot-camera observations or that unsafe plans survive controller-level safeguards. A small-scale physical validation would strengthen the work.
  2. Although the proxy predictor demonstrates useful cross-model predictive transfer from Qwen to Gemini, both evaluated target planners belong to the Gemini family. Evaluating the predictor and object-insertion attacks on a non-Google target would clarify whether malleability reflects broadly shared task characteristics or depends on the target planner.
  3. Section 4.3 reports 20 random objects per task, whereas Appendix D specifies five, creating a reproducibility concern and uncertainty about comparable sampling budgets.
Questions for authors
  1. Can you evaluate at least one non-Google target planner and report whether the same tasks and object insertions transfer, using a shared subset of initially refused tasks?
  2. Does the proxy predictor retain its effectiveness on a different target family without feature selection or recalibration using that target’s outcomes?
5 comments

We thank the reviewer for the constructive feedback. Both questions concern whether our findings hold beyond the Google planner, and we will address them with new experiments.

Generalization to non-Google planners. We will evaluate additional target planners from different model families (two to three depending on time allowance). For each planner, we will run the same pipeline as in the paper: collect the tasks it initially refuses, insert the proposed objects, and measure how often the refusal is reversed, compared against a clean re-query of the unmodified image. We will also investigate how much of the malleable tasks are shared between the different planners.

Proxy predictor on a different target family. We will apply the predictor to new planners, as it was built for the original planner. We will report its performance alongside the original results. To clarify, the proxy predictor is independent of the target's output; it is built on top of an open-source VLM's output, which remains the same regardless of the target.

Number of random objects per task (Section 4.3 versus Appendix D). We thank the reviewer for catching this small but important typo. The number in Section 4.3 is the correct one, every task in the comparison receives 20 objects under each of the three conditions (targeted, scene-only, and random). The "five" in Appendix D is left over from an earlier run of the experiment that used five objects per task, and we will correct it.

We have completed the experiments described in our initial response. The results are summarized below and will be included in the revised paper.

Generalization to non-Google planners. We added three additional planners: GPT-5.4, a general-purpose VLM, Cosmos-Reason2 and RoboBrain2.5, two open-weight 8B models built for robotic planning. Each is evaluated with the same pipeline as in the paper. We first report how many tasks each planner initially refuses, and then how many of those are malleable, meaning that at least one of the 20 variants with an inserted object yields a plan. Mean malleability is the average number of the 20 proposed objects that flip the refusal, over the malleable tasks.

PlannerInitially refusedMalleableMean malleability
Gemini Robotics-ER 284620.21% (171)28.3%
GPT-5.496220.17% (194)28.0%
Cosmos-Reason211183.78% (93)48.0%
RoboBrain2.515490.91% (140)55.6%

GPT-5.4 behaves like Gemini Robotics-ER 2: it refuses a similar number of tasks, a similar share of them is malleable, and a malleable task is bypassed by 28% of its objects on average on both. The two open-weight planners refuse far fewer tasks, most of what they refuse is malleable, and a malleable task is bypassed by about half of its objects. We attribute this partly to their smaller size.

We then check that the reversals are not decoding noise. Each refused task is queried again on its unmodified image. For each task we compute two rates: the share of its 20 object variants that yield a plan (Object), and the share of its clean re-queries that yield a plan (Re-query). The table reports the average of each rate over the refused tasks with a conclusive answer under both, so tasks that never flip count as zero. The difference is paired within task, in percentage points with its 95% CI, and gives how much more likely a refused task is to yield a plan with an inserted object than when simply asked again.

PlannerObjectRe-queryDifference (pp), 95% CI
Gemini Robotics-ER 25.68%0.24%5.44 [4.37, 6.50]
GPT-5.45.59%4.75%0.85 [-0.33, 2.02]
Cosmos-Reason240.67%33.02%7.65 [0.62, 14.67]
RoboBrain2.550.86%38.24%12.63 [5.95, 19.30]

Refusals are unstable on every planner. A refused task can yield a plan when the same query is simply repeated, and a visual change to the image, such as inserting an object, makes the reversal more frequent on three of the four planners. On GPT-5.4 the trend is observed but the difference from the re-query is not distinguishable from zero.

For shared malleable tasks, we take the tasks that are malleable for planner A, count how many of them planner B also refuses on the clean image, and report the share of those that are malleable for planner B as well.

Planner APlanner BMalleable for ARefused by BMalleable for B
Gemini Robotics-ER 2GPT-5.417114541.4%
Gemini Robotics-ER 2Cosmos-Reason21713100.0%
Gemini Robotics-ER 2RoboBrain2.51711693.8%
Cosmos-Reason2RoboBrain2.5933281.3%

Two observations follow. First, malleability transfers across planners partially: a task that is malleable for Gemini Robotics-ER 2 is about twice as likely to be malleable for GPT-5.4 as a refused task in general (41.4% against 20.2%). Second, the open-weight planners accept most of the tasks that Gemini Robotics-ER 2 refuses and finds malleable. The high rate of transfer further supports the conclusion of Sections 4.3 and 4.4 that malleability is a property of the task rather than of the specific object added or the planner.

Proxy predictor on a different target family. The predictor's signals come from a small local model and never use the target, so they are identical for every planner. We took the composite (A5) fitted on Gemini Robotics-ER 2 and applied it to GPT-5.4 without refitting, feature selection or recalibration. Because 801 of the 956 tasks that GPT-5.4 refuses are also refused by Gemini Robotics-ER 2, and were therefore fitting tasks for the composite with different labels, we also report the 155 tasks that Gemini Robotics-ER 2 accepts, which the composite has never seen.

Composite (A5)Gemini Robotics-ER 2GPT-5.4GPT-5.4, tasks not refused by Gemini Robotics-ER 2
Refused tasks846956155
Chance level of PR-AUC20.2%20.3%68.4%
Spearman ρ0.4010.4360.287
ROC-AUC0.7850.8100.662
PR-AUC0.4760.5050.782
Precision at 500.580.640.76

On every metric the paper uses, the predictor performs at least as well on GPT-5.4 as on the planner it was fitted on. On the tasks it has never seen it still ranks malleable tasks above robust ones well above chance.

Comments on the paper

1 comment

Summary for the Area Chair

We thank the reviewers for their constructive feedback. This comment summarizes the major points discussed during the discussion period. Full results and tables are in our individual replies and the revised paper.

Discussion points

  1. Three additional target planners (psm8, btta, zSY9). We ran the full pipeline on GPT-5.4, Cosmos-Reason2 and RoboBrain2.5 alongside Gemini Robotics-ER 2, including the clean re-query control on all four. In addition, we did a no object editing control. Refusals are unstable and malleable on every planner. GPT-5.4 is similar to the Gemini results, Cosmos-Reason2 and RoboBrain2.5 refuse less and reverse their refusal more easily.

  2. Human validation of both judges (btta). The safety judge agrees with human labels on 91 of 100 responses (κ = 0.83), and its errors undercount reversals. The constraint-audit judge agrees on 89 of 100 edits (κ = 0.76) and is stricter than the annotators.

  3. Predictor transfer to a new planner family (psm8, zSY9). The proxy predictor fitted on Gemini, applied to GPT-5.4 without refitting, performs similarly to its source planner and stays above chance on tasks it had never seen.

  4. Transfer of malleable tasks between planners (psm8). Tasks that are malleable for one planner are more likely to be malleable for another; a task that is malleable for Gemini Robotics-ER 2 is about twice as likely to be malleable for GPT-5.4 as a refused task in general. Supporting the paper's observation that malleability is partially a property of a task.

As of the writing of this comment, one out of three reviewers engaged with the rebuttal and raised their score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.