acceptodds
Under review as a conference paper at ICLR 2027

Doubt Before Acting: Assumption-Level Representation Editing for Safe Embodied LLM Planning

Abstract

Embodied agents increasingly use large language models (LLMs) to turn goals into plans with many steps, and an unsafe physical action often cannot be undone. A plan that executes correctly can still be unsafe when it relies on unchecked assumptions about the world, such as a corridor staying clear or a person staying where last seen. To act safely, an agent needs to know whether its plan still works when these assumptions break, and it needs to know this before the first action. We introduce DoubtBeforeAct, an inference-time layer that performs this check. The layer extracts typed assumptions from the scene and uses one catalog of typed operators in two modes. Propose edits the scene to obtain different candidate plans, and Doubt edits the assumptions to build plausible worlds in which they fail. DoubtBeforeAct executes every candidate in these worlds, picks the plan whose worst outcomes are best, and monitors its assumptions during execution. We evaluate on five embodied safety benchmarks and three long-horizon embodied benchmarks against thirteen baselines at the same budget of model calls and executions. DoubtBeforeAct raises safe and successful completion by 8.1 points and long-horizon success by 7.2 points with Qwen3-32B, and by 7.9 to 14.6 points with Qwen3-8B and a frontier model. Test worlds come from assumption classes withheld during development and from a separate authoring pipeline. At equal wall-clock time it still beats the strongest baseline on all eight benchmarks, and as a red teaming tool DoubtBeforeAct finds 205 avoidable failures per 1,000 runs against 104 for the best external generator.

Primary area:
applications to robotics, autonomy, planning
Keywords:
embodied agents, LLM planning, safety, assumption failures, counterfactual worlds, human-robot collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Z99sOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces DOUBTBEFOREACT, an inference-time framework for improving the safety of embodied LLM planning by explicitly examining uncertain assumptions before execution.

The framework extracts task-relevant assumptions from the current scene and uses typed representation-editing operators in two modes. Propose generates alternative candidate plans, while Doubt constructs plausible counterfactual worlds in which the original assumptions may fail. Candidate plans are evaluated across these worlds, and a risk-sensitive selection rule based on Conditional Value at Risk (CVaR) chooses the plan with the best lower-tail performance. A runtime monitor subsequently checks whether the assumptions supporting the selected plan remain valid.

The authors evaluate the framework on five embodied safety benchmarks and three long-horizon planning benchmarks, comparing against thirteen baselines. The paper also includes ablation studies, held-out assumption classes, independently generated test worlds, computational budget controls, and a separate red-teaming evaluation.

Critical strengths
  1. Meaningful safety problem formulation.

The paper addresses an important limitation of embodied LLM planning: a plan that appears valid under the currently observed state may become unsafe when its underlying assumptions fail. Explicitly identifying these assumptions and examining possible violations before execution is a relevant direction for improving embodied-agent safety.

  1. Broad experimental coverage and useful methodological controls.

The evaluation covers multiple embodied safety and long-horizon benchmarks and includes a substantial collection of relevant baselines. The authors also make efforts to evaluate generalization beyond scenarios generated by their own method through held-out assumption classes and independently authored test worlds.

The inclusion of component ablations, equal-wall-clock comparisons, and additional compute-budget controls is valuable. These experiments make the proposed framework more thoroughly evaluated than a simple comparison against unmodified LLM planners.

However, the serious numerical inconsistencies discussed below substantially undermine confidence in the reported empirical conclusions.

Critical weaknesses
  1. Serious numerical inconsistencies across the main results and appendix undermine the reliability of the central experimental claims.

My primary concern is that several reported experimental results cannot be reconciled using the definitions and numerical counts provided in the paper.

First, the main results in Table 1 are inconsistent with the full-method success counts reported in Appendix Table 13.

For example:

  • DESPITE: Table 1 reports 78.1% success, whereas Appendix Table 13 reports 3,779 successes out of 5,220 evaluations, corresponding to approximately 72.39%.
  • EMBODYGUARD: Table 1 reports 79.8%, whereas 2,562/3,420 corresponds to approximately 74.91%.
  • SafeAgentBench: Table 1 reports 69.8%, whereas 2,378/3,750 corresponds to approximately 63.41%.
  • IS-Bench: Table 1 reports 63.2%, whereas 458/805 corresponds to approximately 56.89%.
  • HAZARD: Table 1 reports 66.5%, whereas 672/1,110 corresponds to approximately 60.54%.

These differences are substantial and cannot be explained by ordinary rounding.

Second, Appendix Table 13 contains an additional inconsistency between the reported per-seed success-count differences and the corresponding percentage-point improvements.

For DESPITE, the five reported success-count differences are [46, 71, 51, 68, 56], totaling 292 additional successes. With the reported total evaluation denominator of 5,220, this corresponds to approximately 5.59 percentage points, rather than the reported mean improvement of 8.0 percentage points.

Since Table 13 explicitly defines the per-seed differences using success counts and the number of paired task IDs, these quantities should be reconcilable under a clearly specified aggregation procedure.

Third, Appendix Table 14 reports a safety macro-average of 65.6% and a long-horizon macro-average of 63.8% for the full method. However, directly averaging the corresponding full-method results in Table 1 yields approximately 71.48% and 69.50%, respectively. Both tables describe equal-weight macro-averages over the same five safety and three long-horizon benchmarks.

These inconsistencies affect the primary results rather than peripheral analyses. In particular, the claimed improvements, statistical confidence intervals, and conclusions regarding performance across multiple benchmarks depend on the correctness of these underlying measurements.

I cannot determine from the manuscript whether the discrepancies arise from different experimental populations, undocumented configurations, aggregation errors, or inconsistent result versions. Nevertheless, the current paper does not provide a sufficient explanation.

Until the authors supply a consistent mapping between the reported counts, denominators, experimental conditions, and summary statistics, I cannot regard the headline experimental improvements as reliably established.

This issue is particularly consequential because the paper's main contribution is supported primarily by empirical performance improvements across multiple benchmarks.

  1. Limited independent methodological novelty and unresolved concerns about the interpretation of computational comparisons.

The proposed framework combines several established techniques, including structured assumption representation, counterfactual scenario generation, robust planning, CVaR-based selection, and runtime monitoring.

The authors acknowledge that CVaR and scenario-based robust optimization are established methods, while the typed editing operators are adapted from the prior TROPE framework. The main methodological contribution therefore appears to lie in integrating these components into an embodied planning pipeline rather than introducing a fundamentally new risk-sensitive planning or uncertainty-reasoning mechanism.

Such integration could still constitute a meaningful contribution if its independent advantages were demonstrated convincingly. However, the numerical inconsistencies in the main results currently prevent a reliable assessment of the claimed gains.

I also have concerns about the interpretation of the matched-compute evaluation.

The headline protocol allocates five LLM calls and 80 execution evaluations to every method. However, Appendix Table 18 distinguishes decision-visible work from accounting-only operations that cannot affect the returned decision.

Under this accounting, DOUBTBEFOREACT uses all 80 evaluations for decision-making, whereas ReAct, Tree of Thoughts, SafeMem, and RoboSafe use 14, 38, 20, and 22 decision-visible evaluations, respectively.

The authors explicitly disclose this distinction and provide additional controls, including matched-native-use, equal-wall-clock, and reduced-execution comparisons. These efforts are appreciated, and I do not consider the nominal-budget issue alone sufficient to invalidate the study.

Nevertheless, equal nominal resource consumption should not be interpreted as equal effective decision-making computation. The central performance claims should be supported by clearly identified, consistent results under the relevant effective-budget conditions.

Finally, the approach relies on generating plausible counterfactual worlds and evaluating plans using symbolic or simulator-backed execution. Although this is appropriate for the studied benchmarks, the ability of the resulting planner to handle omitted assumptions, inaccurate transition models, or physically invalid counterfactual scenarios remains insufficiently established.

Overall, the proposed combination is potentially useful, but its independent methodological contribution and empirical advantages are not convincingly demonstrated in the current submission, particularly given the unresolved inconsistencies in the primary results.

Questions for authors

Q1. Can the authors explain and reconcile the numerical discrepancies between Table 1 and Appendix Table 13?

For example, Table 1 reports 78.1% success on DESPITE, while Table 13 reports 3,779/5,220 full-method successes, corresponding to approximately 72.39%.

Similar discrepancies occur across all five safety benchmarks.

What are the exact evaluation populations, denominators, model configurations, and aggregation procedures underlying these two tables? Do they correspond to different experiments or versions of the evaluation?

Please provide the underlying per-task, per-seed success records and the aggregation procedure used to generate the reported results.

Q2. Why are the reported success-count differences and macro-averages inconsistent with their stated definitions?

Appendix Table 13 reports five DESPITE success-count differences totaling 292, which corresponds to approximately 5.59 percentage points over 5,220 evaluations, rather than the reported 8.0-point mean improvement.

Similarly, Appendix Table 14 reports full-method safety and long-horizon macro-averages of 65.6% and 63.8%, while the corresponding results in Table 1 imply approximately 71.48% and 69.50%.

Can the authors clarify whether these differences arise from undocumented evaluation subsets, alternative aggregation procedures, or errors in the reported tables?

If the primary tables require correction, how would this affect the reported improvements, paired confidence intervals, statistical significance tests, and conclusions?

Q3. What is the independent advantage of the proposed assumption-level counterfactual editing mechanism under comparable effective computation?

The framework combines established scenario-based robust planning and CVaR optimization with typed editing operators adapted from prior work.

Can the authors clarify which component provides the principal methodological advancement beyond conventional uncertainty-aware planning?

In addition, given the distinction between nominal and decision-visible computation in Appendix Table 18, can the authors identify the results that most directly establish superiority under comparable effective planning resources, with paired uncertainty estimates?

Answers to these questions, particularly Q1 and Q2, would materially affect my assessment of the submission.

4 comments

(1/2)

Thank you for separating the numerical, methodological, and computational concerns. We agree that the appendix discrepancies were substantive. The revision replaces the inconsistent appendix summaries, supplies seed-level counts for both methods, and adds paired intervals for each resource control.

Q1. Evaluation populations, versions, and aggregation. The submitted manuscript combined the main results of the final run with appendix tables and comparator labels from an earlier result version (general response). The old counts are not a different rounding or weighting of the main results, and we withdraw them.

  • The primary condition is Qwen3-32B on assumption-break tasks derived from the eight benchmarks, with five paired seeds and ConceptAgent as the comparator. Both methods use the same task IDs and seeds.
  • The task counts per seed are DESPITE 1044, EMBODYGUARD 684, SafeAgentBench 750, IS-Bench 161, HAZARD 222, PARTNR 1000, EAI 1000, and EB-ALFRED 300.
  • Native task subsets are reported separately (Table 49) and are not pooled into Table 1.
  • Tables 13 and 45 give the corrected totals, the per-seed differences, and the per-method seed counts. For example, DESPITE has 4,077/5,220 successes for DBA and 3,659/5,220 for ConceptAgent, which reproduces 78.10% and 70.10%. The other safety totals for DBA (2,729/3,420, 2,618/3,750, 509/805, and 738/1,110) reproduce 79.80%, 69.81%, 63.23%, and 66.49%.
  • A benchmark score is the mean of five seed-level percentages. Because the denominators are equal, it also equals 100 × total successes / (5 × tasks). Macros average benchmark percentages with equal weights, separately over the five safety and the three long-horizon benchmarks. Safety uses safe-and-successful completion, and long-horizon uses task success.
  • Table 45 lets every displayed rate, seed difference, and macro be recomputed directly from the paper.

Q2. Seed differences, macros, intervals, and significance.

  • The old DESPITE row summed to 292 and was incompatible with an 8.0-point gain. The corrected differences are [96, 86, 59, 83, 94], which sum to 418, so the gain is 8.01 points. The previous percentage row was not computed from these counts. The revised row is, and it shows the actual seed-to-seed variation (9.20, 8.24, 5.65, 7.95, and 9.00 points).
  • The corrected macros are 71.49% (safety) and 69.50% (long-horizon), replacing 65.6% and 63.8%. ConceptAgent's macros are 63.41% and 62.27%, giving gains of +8.08 and +7.23. Tables 14 and 32 are synchronized, and Table 34 uses the corrected failure total of 3,634.

Table 11 reports the updated primary inference:

BenchmarkΔ, ppPaired 95% CIHolm-adjusted p
DESPITE+8.01[5.58, 10.43]7.97e-10
EMBODYGUARD+6.99[3.58, 10.40]0.000238
SafeAgentBench+8.72[5.79, 11.65]3.45e-08
IS-Bench+8.20[1.32, 15.08]0.0389
HAZARD+8.47[0.47, 16.47]0.0389
PARTNR+7.90[5.45, 10.35]1.97e-09
EAI+7.20[3.77, 10.63]0.000191
EB-ALFRED+6.60[1.55, 11.65]0.0312
  • We use 10,000 paired task-cluster bootstrap resamples, which keep all five seeds of a task together. The intervals are Δ ± 1.96 times the bootstrap SE. The p-values are two-sided normal-theory values with Holm correction across the eight benchmarks. Macro intervals resample tasks within each benchmark and keep the equal benchmark weights fixed.
  • The macro gains are +8.08 [5.71, 10.44] for safety, +7.23 [5.05, 9.41] for long-horizon, and +7.76 [6.08, 9.44] over all eight benchmarks.
  • The direction of the headline result is unchanged, and all eight adjusted p-values are below 0.05. The intervals for IS-Bench and HAZARD are wide, however, and we no longer claim uniformly tight intervals or p < 0.001 everywhere.

(2/2)

Q3. Independent contribution and comparable effective resources.

  • We frame the contribution as an embodied integration, not as a new CVaR objective or a new heavy-tailed distribution. TROPE supplies the representation schema, the operator families, and edit-size sampling. DBA uses scene assumptions as an explicit interface that links candidate generation, counterfactual world construction, plausibility checks, plan selection, and runtime monitoring.
  • With Gaussian-W1 sampling, DBA remains 4.8 to 5.9 points above ConceptAgent on every benchmark (Table 1), roughly two thirds of the full gain.
  • Table 48 removes each component under Gaussian-W1. Doubt contributes +5.31 [3.50, 7.12] on DESPITE, +4.00 [2.07, 5.93] on PARTNR, and +4.20 [0.64, 7.76] on EB-ALFRED. The filter and the monitor also have positive intervals on DESPITE and PARTNR, while on EB-ALFRED their intervals include zero.

We agree that the nominal limit of 5 calls and 80 evaluations does not equalize decision-making work. The revised Table 18 reports the calls and evaluations that each method actually completes in the final run. The decision-visible counts you quote (14/38/20/22) came from the superseded appendix. Three controls address different resource questions, each with paired intervals:

  • Under the time cap (Table 20), DBA is limited to ConceptAgent's per-benchmark planning time (mean cap 9.98 s, actual DBA mean 9.41 s). The gains are +7.00 [4.65, 9.34] on the safety macro, +6.31 [4.11, 8.52] on the long-horizon macro, and +6.74 [5.06, 8.42] over all eight benchmarks. Seven of the eight benchmark intervals exclude zero, and HAZARD's is [−0.56, 15.34]. This is the most direct comparison under equal measured resources.
  • At its native operating point (Tables 22 and 50), ConceptAgent uses 9.12 calls and 16.16 s, versus 5 calls and 11.23 s for DBA. DBA's gains are +8.30 [5.74, 10.86] on PARTNR, +6.20 [2.77, 9.63] on EAI, and +5.67 [0.41, 10.92] on EB-ALFRED, with a macro of +6.72 [4.48, 8.97]. This removes the five-call restriction on tree search.
  • With reduced evaluations (Tables 28 and 50), m = 5, DBA uses 20 evaluations, against RoboSafe's 14.14 completed evaluations. The gains are +6.80 [4.10, 9.50] on DESPITE, +5.10 [2.23, 7.97] on PARTNR, and +6.33 [1.21, 11.45] on EB-ALFRED, with a macro of +6.08 [3.93, 8.23]. The number of evaluations is closer but still not equal.

These controls do not establish identical FLOPs or token use across methods, and we do not claim that they do.

Omitted assumptions, transition errors, and invalid worlds.

  • DBA can only stress-test the assumptions it extracts, using the execution interface it is given. An omitted premise cannot be recovered by changing the sampler, and the monitor checks only represented assumptions.
  • Schema validation, plausibility filtering, held-out assumption classes, and independently authored test worlds address specific failure modes. They do not prove physical validity or correct transition models.
  • The Limitations section now states this, and we do not infer real-robot safety from benchmark results.

Thank you again for the constructive review. You noted that answers to Q1 and Q2 would materially affect your assessment. We hope the corrected counts, intervals, and controls above provide them, and we would be grateful if you would consider revisiting your score. We are happy to clarify anything further.

Reviewer_Z99sOpenReview ↗

Assessment of the Rebuttal and Revised Manuscript

Thank you for the detailed rebuttal and substantial revisions. I appreciate the authors' transparency in acknowledging the numerical inconsistencies and providing corrected results with additional experimental controls.

The revised per-seed counts are consistent with the main results, and the new paired confidence intervals, Gaussian-W1 comparisons, and computational controls considerably strengthen the empirical evidence. I therefore consider my original concern regarding numerical reliability substantially addressed.

Nevertheless, two concerns remain.

1. Methodological novelty and generalization. DBA primarily integrates existing representation-editing techniques, counterfactual evaluation, CVaR-based planning, and runtime monitoring. While the integration is meaningful, its independent methodological novelty remains limited. Moreover, the reported gain decreases from +7.76 percentage points on assumption-break tasks to +2.01 points on unmodified native tasks, with the native long-horizon confidence interval including zero. This limits the generality of the conclusions.

2. Statistical clustering. Some benchmark-native tasks generate multiple derived assumption-break instances. Could the authors clarify whether the paired bootstrap clusters by the original native task ID or by individual derived instances? If the latter, a native-task-level sensitivity analysis would help assess the impact of within-task dependence.

Overall, the rebuttal substantially improves the paper and resolves my most serious original concern. However, I remain unconvinced that the methodological novelty and generalization evidence are sufficient for acceptance.

I have therefore increased my rating from 1 (Clear Rejection) to 2 (Weak Rejection).

Thank you again for the constructive response.

Thank you for the careful reassessment and for raising your score. We answer the two remaining points.

Statistical clustering. The bootstrap in Table 11 resampled derived assumption-break instances, with all five seeds of an instance kept together. Following your suggestion, we reran it clustered by native task ID, keeping all derived instances of a native task and all five seeds together:

EffectNative-task-clustered 95% CI
Safety macro+8.08 [5.49, 10.67]
Long-horizon macro+7.23 [4.99, 9.48]
All-8 macro+7.76 [5.94, 9.58]
HAZARD+8.47 [−1.17, 18.11]

The macro conclusions are unchanged. The HAZARD interval now includes zero, so we no longer claim a significant per-benchmark effect on HAZARD. The next version will report the native-task-clustered intervals as the primary ones and revise the per-benchmark claims accordingly.

Novelty and generalization. We agree that the contribution is an integration, and the revised paper states it that way. On generality, DBA is designed for the case where an assumption the plan relies on breaks. In unmodified native tasks no break is injected, so the counterfactual check has less to act on, and a smaller gain is the expected outcome. We report it as measured: +2.01 [0.66, 3.36] overall, with positive intervals on DESPITE and PARTNR and an inconclusive long-horizon macro, and we do not extend the assumption-break gains to native tasks.

Thank you again for the constructive engagement.

Reviewer_AUUQOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DOUBTBEFOREACT (DBA) wraps an unchanged LLM planner. One call extracts a typed scene representation with explicit unverified assumptions (eight classes). A TROPE-derived operator catalogue is used in two modes: Propose edits the scene to obtain alternative plans (four candidates), and Doubt edits assumptions to build 19 counterfactual worlds, with α-stable edit sizes (α = 1.6) and a retrieval + NLI plausibility filter over an incident corpus. Each candidate is executed in every world (80 executions); the plan with the best mean over its four worst utilities (CVaR) is chosen, a fallback fires below 0.5, and a monitor re-doubts during execution. Evaluation: five safety and three long-horizon benchmarks, each instantiated as (task, injected assumption break) items from a leave-two-classes-out suite, 13 baselines under a shared budget of 5 LLM calls and 80 executions, three backbones. Reported gains over the strongest baseline (ConceptAgent) are 7.0–8.7 points on every benchmark, plus equal-wall-clock, native-use, radicality and component ablations, a red-teaming comparison (205 vs 104 failures per 1,000 runs) and a PARTNR case study.

Critical strengths

Coherent method. Assumptions are named, typed, traced to plans, broken by checked operators and monitored; CVaR over sampled worlds is the right criterion for "best in the observed scene but fragile". Sec. 3.2 is honest about what is borrowed. This is why a corrected version would be worth having. Careful protocol, as described. Held-out assumption classes, test worlds authored without the method's operators, decision-visible vs accounting-only budgets (Tables 18, 27), equal wall-clock (Table 20), native operating points for some baselines (Table 22), paired bootstraps with Holm correction, radicality sweep with intervals (Table 31), blinded realism audit (Table 8).

Critical weaknesses

The appendix contradicts the headline numbers, systematically and in one direction. The counts under Table 13 (e.g. DESPITE 3779/5220, IS-Bench 458/805) give full-DBA scores of 72.4 / 74.9 / 63.4 / 56.9 / 60.5, not Table 1's 78.1 / 79.8 / 69.8 / 63.2 / 66.5. Their macro average is exactly the 65.6 that Tables 14 and 32 report for "Full DBA", and their misses sum to exactly Table 34's 4,456. The comparator implied by the counts (66.8 / 69.1 / 56.9 / 51.8 / 53.5) equals RoboSafe's and SafeMem's Table 1 scores, not ConceptAgent's; the implied gains are 5.1–7.0 points, not 7.0–8.7. Per-seed count differences also vary far more than Table 13's percent rows (IS-Bench [2, 10, 9, 13, 7] of 161 = 1.2–8.1%, listed as 8.0–8.4). Table 2B / Table 8 report archive coverage of 91.2% (DBA) and 78.4% (Gaussian) and say it is computed from Table 9, but Table 9 gives 529/800 = 66.1% and 461/800 = 57.6%; Figure 6's "+8.5" matches Table 9. The other five generators agree across both tables. Leftovers point the same way: "nine original baselines" and a 7.5 s cap macro in Appendix G.3/G.5 versus thirteen baselines and 10.0 s in the main text. Every quantitative claim in the abstract depends on which set is correct. I do not assume intent, but this must be resolved with the task-level logs the reproducibility statement promises. Even at face value, the comparison is tilted toward the method. Every headline item is a native task with an injected break from the authors' own eight-class taxonomy (Appendix G.1); no result is reported on unmodified benchmark tasks, yet the main text calls them "five embodied safety benchmarks". ConceptAgent, a tree-search method and the strongest baseline, is only ever run under the 5-call cap and is missing from the native-operating-point control (Table 22). The heavy-tail claim rests on the Gaussian control, but α = 1.8 / 1.4 / 1.2 are statistically indistinguishable from α = 1.6 (Table 31) while the Gaussian (α = 2) is 6.7 DESPITE points below α = 1.8 (Table 30); the control must differ in more than tail shape (scale matching). The red-team margin is fragile too: removing any single incident-source family cuts DBA from 205 to 139–145 failures/1k while RedVLA stays at 101–103 (Table 44); under a blinded filter the ratio drops from 2.0× to 1.5× (Table 24). Reported latencies are hard to credit: 80 simulator executions plus five Qwen3-32B calls in 13.6 s on Habitat (Table 19); five sequential gpt-5.6-sol high-effort calls plus 80 executions in 7.6 s (Table 21). If the executions are symbolic rollouts over R rather than simulator runs, say so.

Questions for authors

(Clarification) Which run produced Table 1, and which produced the Table 13 counts, Table 14, Table 32 and Table 34? Why do the counts imply RoboSafe/SafeMem as comparator and DBA scores 5–6 points lower on every safety benchmark? Is Table 2B's 91.2% / 78.4% or Table 9's 66.1% / 57.6% correct? Please provide per-seed success counts for DBA and ConceptAgent on all eight benchmarks. (Clarification) Are the 80 executions simulator rollouts or symbolic executions over R, and how are 13.6 s / 7.6 s per plan achieved with five sequential LLM calls? What does DBA score on unmodified native tasks, and what does ConceptAgent score at its native operating point? (Suggestion) Run an α = 2 point through the same scale-matching pipeline as the stable settings so the heavy-tail claim is tested within one family.

5 comments

(1/3)

Thank you for tracing the inconsistencies in such detail. You were right that the submitted appendix did not support the main results as written. In the revision, the numerical summaries come from the final run. We also separate assumption-break tasks from native tasks, and we add the calibration and comparator controls you asked for.

1. Which run produced which table, comparators, and seed counts.

  • Table 1 comes from the final run. The submitted Table 13 counts and Tables 14, 32, and 34 came from earlier runs in which DBA used a single Doubt world per decision instead of 19, and were not updated before submission. This is why they imply DBA scores 5–6 points lower: they fall between the observed-world-only ablation and the m = 5 condition in Table 10 (DESPITE 72.4% versus 70.0% and 73.6%).
  • Some appendix analyses still used the comparator convention of the original nine-baseline pool, which is why the counts imply RoboSafe and SafeMem. In that pool, ConceptAgent had not yet been included. The revised Table 12 records the development-set selection, which picks ConceptAgent on all eight benchmarks.
  • We withdraw the earlier counts and the mislabeled comparisons as support for the headline claims.

The primary condition uses Qwen3-32B, five paired seeds, assumption-break task populations, and ConceptAgent as the frozen comparator. Both methods run under common caps of five model calls and 80 planning evaluations. Actual completed work is reported separately (Tables 18 and 27).

BenchmarkTasks/seedDBA successesConceptAgent successes
DESPITE10444077/52203659/5220
EMBODYGUARD6842729/34202490/3420
SafeAgentBench7502618/37502291/3750
IS-Bench161509/805443/805
HAZARD222738/1110644/1110
PARTNR10003595/50003200/5000
EAI10003710/50003350/5000
EB-ALFRED300936/1500837/1500
  • Table 45 gives the success count of each method at every seed, with seed means and sample SDs. Table 13 gives the paired seed differences. For DESPITE these are [96, 86, 59, 83, 94], which sum to 418, and 100·418/5220 = 8.01 points.
  • The revised percentage rows show the actual seed variation. For IS-Bench the row is now 11.2, 8.7, 0.6, 9.3, and 11.2 (counts [18, 14, 1, 15, 18]), replacing the 8.0–8.4 you noted.
  • The macros are 71.49% (safety) and 69.50% (long-horizon), and the safety failure total is 3,634.
  • Coverage is 730/800 (91.25%) for full DBA and 627/800 (78.375%) for legacy Gaussian. Table 9 and Figure 6 are corrected.
  • The "nine original baselines" text and the 7.5 s cap macro are removed. The comparator's mean time cap is 9.975 s.

(2/3)

2. What the 80 evaluations are, and latency.

  • The 80 planning evaluations are four candidate plans, each evaluated in 20 worlds (the observed world plus 19 accepted Doubt worlds). They are not 80 complete benchmark episodes. As stated in Section 3.2, plans are executed through one symbolic interface over R, which applies PDDL/BDDL transitions directly. On simulator benchmarks, including PARTNR, the simulator executes the plan without rendering, and the resulting state is converted back to typed predicates.
  • The five model calls are one extraction call, one call for the unedited plan, and three candidate-generation calls.
  • The timer runs from the moment the common planner-visible state is available until a plan or fallback is returned. It includes model calls, retrieval, edits, filtering, and candidate/world evaluation. It excludes the evaluator-only rollout. Simulator startup and cache warmup are excluded in the same way for every method (Appendix K).
  • For Qwen3-32B, Table 19 reports end-to-end planning latency, which is 13.6 s on PARTNR. For the frontier model, Table 21 reports a different quantity: the client-side wait for planner-model API responses only, with concurrent calls overlapping. World generation, the 80 evaluations, and local processing are excluded, and the figure is averaged with equal weights over all eight benchmarks. For DBA it is 6.09 s, superseding the earlier 7.6 s. The submitted caption did not state this definition clearly, and the revised caption now does.

How the latency is achieved. The low latency comes from parallelism, not from fewer calls.

  • Qwen3-32B is served with vLLM on 12 NVIDIA H200 (141 GB) GPUs. All methods, including every baseline, run on the same hardware and the same worker pool, so the equal-time comparison (Table 20) gives no method a hardware advantage.
  • After the extraction call, the unedited-plan call and the three Propose calls are issued concurrently. Doubt world generation runs in parallel with them, because it edits the assumptions H and does not depend on the candidate plans. The serial LLM path is therefore two call rounds.
  • On PARTNR, the 80 candidate/world evaluations run as headless Habitat rollouts on 14 parallel workers, at about 1.05 s per rollout. That is 6 rounds, or roughly 6.3 s. The remaining ~7.3 s of the 13.6 s are the two LLM rounds, Doubt, and selection.
  • For gpt-5.6-sol, the four plan calls are also issued concurrently, so DBA's API wait covers two call rounds. This is comparable to Zero-shot CoT's 2.06 sequential calls, which is why the API wait times in Table 21 are similar although DBA uses about three times more tokens. Table 21 is not an end-to-end latency, and we do not present it as one.

Tables 38 and 18. Table 38 lists maximum settings, and Table 18 reports the calls actually completed:

  • Zero-shot CoT draws samples one at a time and stops at the first valid plan, which gives 2.06 completed calls on average.
  • KnowNo generates its three candidate options in a single call and then runs up to two help checks, which gives 2.54.

The Table 38 caption now says this.

3. Unmodified native tasks and ConceptAgent's native operating point. The headline results are now explicitly described as results on assumption-break tasks derived from the benchmarks.

Table 49 reports unmodified native task subsets with their own denominators: 1044, 402, 750, 161, 70, 1000, 438, and 300 tasks per seed, in the order of the table above. On these subsets, the safety macro gain is +2.30 [0.41, 4.20], the long-horizon gain is +1.53 [−0.17, 3.23], and the eight-benchmark gain is +2.01 [0.66, 3.36]. Only DESPITE and PARTNR have individual intervals that exclude zero.

The benefit on native tasks is therefore smaller, and the long-horizon macro is inconclusive. The abstract, introduction, conclusion, and limitations now say so.

ConceptAgent's native operating point is reported in Tables 22 and 50. At this operating point, ConceptAgent uses 9.12 calls and 16.16 s on average, versus 5 calls and 11.23 s for DBA. Its scores, and DBA's paired gains over it, are:

BenchmarkConceptAgent (native)DBAPaired gain
PARTNR63.671.9+8.30 [5.74, 10.86]
EAI68.074.2+6.20 [2.77, 9.63]
EB-ALFRED56.762.4+5.67 [0.41, 10.92]
Macro (3 benchmarks)62.869.5+6.72 [4.48, 8.97]

(3/3)

4. α = 2 through the common calibration. We agree that the legacy Gaussian comparison confounded the distribution with the calibration criterion. Table 46 adds Gaussian-W1 (α = 2), calibrated with the same W1 criterion as the stable settings. Relative to it, α = 1.6 improves the safety macro by +2.78 [1.47, 4.09], the long-horizon macro by +1.99 [0.84, 3.14], and the eight-benchmark macro by +2.48 [1.56, 3.41].

Four of the eight individual intervals exclude zero. Results under the legacy median-matched Gaussian keep their own label.

Gaussian-W1 DBA itself remains 4.8 to 5.9 points above ConceptAgent on every benchmark (Table 1). Most of the gain therefore does not depend on the stable tail.

As you noted, α = 1.6 is not singled out by the sweep. In the revised Table 31, α = 1.8, 1.4, and 1.2 are each about 0.9 points higher, with paired intervals that exclude zero, and all of them lie above Gaussian-W1. The heavy-tail effect is therefore robust across the stable range rather than tied to one value. We keep 1.6, the setting inherited from TROPE, rather than switching to a value picked from test results, and we no longer suggest that it is optimal.

5. Sensitivity of the red-teaming results. We agree that the 205-versus-104 result depends on the incident corpus and the filter. Tables 44 and 24 were regenerated with the rest of the appendix, so the values you quoted (DBA 139–145, RedVLA 101–103, and the 1.5× ratio under the blinded filter) are superseded.

  • In Table 44, with one source family removed (consumer, occupational, or robotics), DBA/RedVLA rates are 141/86, 123/82, and 127/74 per 1,000 runs. RedVLA's rate also falls, because the same filter is applied to all generators.
  • In Table 24, the 400-run-per-generator audit gives 217.5/107.5 under the default filter and 182.5/85.0 under the alternative filter.

DBA finds more avoidable failures than RedVLA in every condition, with ratios from 1.5× to 2.1× in point estimates. Its absolute rate does fall substantially when a source family is removed. We report this dependence and do not claim an invariant twofold advantage.

Thank you again. Your review substantially improved the paper. We hope the revision resolves the concerns you raised, and if so, we would be grateful if you would consider revisiting your score. We are happy to clarify anything further.

Reviewer_rNY8OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DoubtBeforeAct, an inference-time layer for LLM-based embodied planners. It first extracts typed assumptions from the scene. Then, it edits the scene to obtain different candidate plans and edits the assumptions to build plausible worlds where they fail. Every candidate is executed in every world, and the plan whose worst outcomes are most acceptable is selected. The paper reports stable gains of the method over thirteen baselines on eight embodied benchmarks.

Critical strengths
  • The research question of improving the reliability and safety of embodied planning when assumptions can fail is significant.
  • The experiments are comprehensive and the evaluation is well designed.
Critical weaknesses
  • Inaccurate methodological contribution attribution relative to TROPE. The representation schema, eight structural operators, and the α-stable edit size 1.6 all match the cited TROPE’s public implementation. Yet the paragraph “What is established and what is new” in Sec. 3.2 lists heavy-tailed edit sizes as a contribution of this work. The genuinely new elements are the embodied assumption taxonomy, using edited representations as evaluation worlds with CVaR selection, the incident-based plausibility filter, and the monitor.

  • The new components’ contribution is not demonstrated by experiment results. The DBA, Gaussian variant in Table 1 keeps every component above except the heavy tail. It beats the best baseline method ConceptAgent by just 0.2–1.8 points across the eight benchmarks, so nearly all of the reported gain comes from the inherited component.

  • Result inconsistencies affect main claims. The coverage values of DBA’s two variants in Table 2B (91.2/78.4) do not match Table 9 (66.1/57.6). The success counts below Table 13 (e.g., 3779/5220 = 72.4% on DESPITE versus 78.1% in Table 1) imply 5–7 points lower scores for the full DBA on all five safety benchmarks. Therefore, it is unclear which numbers the main conclusions rest on.

Questions for authors
  • Please check the discrepancies in Weakness 3.
  • Please correct the attribution of the heavy-tailed edit sizes and explain the mechanism behind the gap between the Gaussian and α-stable variants.
  • Could you provide evidence to show the components new to this paper contribute substantially on their own, independent of the heavy tail?
3 comments

Thank you for the review. You raised three problems: the appendix numbers were inconsistent, the attribution of heavy-tailed edit sizes was wrong, and there was no evidence that the new components help without the heavy tail. We agree with all three. The revision addresses each one, and the general response lists every changed table.

W3 / Q1. Discrepancies. The submitted appendix was not regenerated from the final run behind Table 1 (see the general response). The differences were not due to rounding, and we withdraw the superseded values.

  • In the revision, Tables 13 and 45 reproduce Table 1 from integer counts. On the five safety benchmarks, DBA succeeds on 4,077/5,220, 2,729/3,420, 2,618/3,750, 509/805, and 738/1,110 tasks (78.1, 79.8, 69.8, 63.2, and 66.5%). ConceptAgent is the comparator on all eight benchmarks, and on DESPITE it succeeds on 3,659/5,220 (70.1%). Table 45 gives the counts of both methods for every seed.
  • Full DBA occupies 730/800 cells (91.25%) and the legacy Gaussian control 627/800 (78.375%), each from the same 4,000-run campaign. Tables 2, 8, and 9 and Figure 6 now use these counts. The old 529/800 and 461/800 counts and the "+8.5" annotation are removed.
  • Table 11 gives the updated inference: +8.08 [5.71, 10.44] on the safety macro and +7.23 [5.05, 9.41] on the long-horizon macro. All eight Holm-adjusted p-values are below 0.05. We no longer claim that all are below 0.001.

W1 / Q2. Attribution and the Gaussian/stable gap. Section 3.2 now states the following.

  • The typed representation schema, the eight structural operator families, and heavy-tailed edit-size sampling (including α-stable sizes with α = 1.6) are adapted from TROPE.
  • CVaR and scenario-based robust planning are established methods.
  • Heavy-tailed edit sizes are no longer listed as our contribution.

Our contribution is their embodied integration. DBA extracts scene assumptions, uses assumption breaks as evaluation worlds for candidate plans, filters these worlds against incident records, and monitors the assumptions of the selected plan during execution.

On the mechanism, the old comparison was confounded. The original Gaussian control matched the median accepted structural distance, while the stable sampler was calibrated with a W1 criterion. We added a Gaussian control calibrated with the same W1 criterion, Gaussian-W1 with α = 2 (Table 46).

  • Relative to Gaussian-W1, α = 1.6 gains +2.78 [1.47, 4.09] on the safety macro and +1.99 [0.84, 3.14] on the long-horizon macro. Four of the eight per-benchmark intervals exclude zero.
  • In the proposal audit (Table 47), median accepted distances are similar (0.272 and 0.267), but α = 1.6 has a longer upper tail (99th percentile 0.739 versus 0.646). This is consistent with the stable sampler sometimes accepting larger assumption breaks, which change the lower tail of plan utilities. We present it as consistent evidence, not as proof of the mechanism. The share of accepted edits with K ≥ 3 is not higher (36.1% versus 36.7%), so we do not attribute the gain to more compound edits.
  • We do not claim that α = 1.6 is optimal (Table 31).

W2 / Q3. Contribution of the new components without the heavy tail. Two results address this question directly.

(i) Without the heavy tail, DBA still clearly beats the strongest baseline.

  • Gaussian-W1 DBA scores 75.3, 77.7, 67.0, 60.0, and 63.5 on the safety benchmarks and 68.8, 72.6, and 61.1 on the long-horizon benchmarks (Table 1).
  • That is 4.8 to 5.9 points above ConceptAgent on every benchmark: +5.3 on the safety macro and +5.2 on the long-horizon macro, or roughly two thirds of the full gain (+8.1 and +7.2). The stable tail adds the remaining +2.8 and +2.0.
  • The "0.2–1.8 points" you noted came from comparing ConceptAgent with the legacy, median-matched Gaussian control.

(ii) Table 48 removes each new component under both samplers. The contributions under Gaussian-W1 are:

Component retainedDESPITEPARTNREB-ALFRED
Doubt+5.31 [3.50, 7.12]+4.00 [2.07, 5.93]+4.20 [0.64, 7.76]
Plausibility filter+2.80 [1.59, 4.00]+4.30 [2.84, 5.76]interval includes zero
Monitor+1.92 [0.70, 3.14]+1.60 [0.26, 2.94]interval includes zero

We do not claim that every component helps significantly on every benchmark, and we do not claim a tested sampler-by-component interaction. On EB-ALFRED, the TROPE-style Propose-only diagnostic remains stronger than full DBA (63.9 versus 62.4, Table 10), and the text now says so.

We hope these corrections and controls address your three concerns. If they do, we would be grateful if you would consider revisiting your score. We would be glad to answer any remaining questions.

Thank you for the detailed response. The attribution in Sec. 3.2 is now accurate, and the tables are internally consistent. However, my main concerns remain.

Provenance of results. The response says the submitted appendix came from an earlier run. In that run, full DBA scored 5–7 points lower on all five safety benchmarks. That gap is about the size of the claimed improvement over ConceptAgent. Moreover, the response does not say what changed between the two runs, whether test results were seen in between, or what ConceptAgent scored in the earlier run. Archive coverage also changed substantially for the same 4,000-run campaign. Without an account of these differences, I cannot tell which results the conclusions rest on.

The new Gaussian-W1 control. This control was introduced after the original control showed only a marginal gain. Both Gaussian controls use α=2, and γ is chosen from the same 7-point grid. The two have nearly identical accepted-distance medians, but they differ by 3.2–5.1 points on every benchmark. In addition, the Gaussian-W1 vs. ConceptAgent comparison has no confidence intervals. Given the bootstrap SEs in Table 11, the gains on IS-Bench and HAZARD are unlikely to be significant.

Contribution beyond TROPE. Under Gaussian-W1, the filter and monitor add 1.6–4.3 points on three benchmarks, and several intervals include zero. On EB-ALFRED, TROPE-style Propose-only outperforms full DBA. A direct TROPE baseline on all eight benchmarks would be needed to establish the size of the new contribution.

I appreciate the substantial revision, and I have raised my rating from 1 to 2 to reflect this. However, given the remaining concerns above, I continue to recommend rejection.

Thank you for reading the revision carefully and for raising your score. We answer the three remaining points directly.

Provenance of results.

  • The earlier runs differed from the final run in one setting: DBA evaluated each candidate in a single Doubt world per decision instead of 19. The final value m = 20 follows from the protocol budget, since 80 evaluations over four candidate plans give 20 worlds including the observed one, and it was fixed before we saw any test results from the earlier runs. The earlier scores lie between the observed-world-only ablation and m = 5 in Table 10, as the m sweep predicts.
  • ConceptAgent was not run in the earlier series. It was added when the baseline pool was extended for the final comparison, so there is no earlier ConceptAgent score.
  • The coverage campaign itself did not change. The submitted Table 9 and Figure 6 came from an intermediate summary of the same 4,000-run campaign. The full export gives 730 and 627 occupied cells, which are the values the submitted Table 2B already reported (91.2% and 78.4% are 730/800 and 627/800). Only Table 9 and Figure 6 were stale.
  • We agree that the paper should document the earlier configuration and its results, and the next version will do so.

The two Gaussian controls. They are not the same sampler in every fold. Both use α = 2 and the same seven-point grid, but median matching and the W1 criterion choose different scales in two of the four folds:

FoldLegacy γGaussian-W1 γ
F10.500.50
F20.500.71
F30.710.71
F40.711.00

In F2 and F4, W1 selects the next grid value, and the W1 distance to the authored development perturbations drops from 0.0270 to 0.0073 and from 0.0271 to 0.0065. Median matching therefore picks a scale about 1.4 times smaller than the distributional fit in half of the folds.

For Gaussian-W1 versus ConceptAgent, the same paired task-cluster bootstrap gives +5.30 [2.99, 7.60] on the safety macro and +5.24 [3.04, 7.45] on the long-horizon macro. We agree that the per-benchmark differences on IS-Bench and HAZARD are unlikely to be significant on their own, so we make this claim at the macro level only. The next version will report these intervals.

Contribution beyond TROPE. We agree that a direct TROPE baseline on all eight benchmarks is the right test of the incremental contribution. We cannot run it before the discussion closes, and we will add it. Until then, the evidence for the new components is the within-pipeline ablation in Table 48, reported with its limits, including EB-ALFRED, where the Propose-only diagnostic is stronger.

Thank you again for the constructive engagement.

Comments on the paper

1 comment

We thank the reviewers for their careful reading. All three reviewers found that the submitted appendix did not match the main results. They were right, and we take responsibility for it. Below we explain the cause, list every table that changed, and describe the new controls and the claims we have narrowed. Changes in the revised PDF are marked in blue.

Cause. The appendix tables in the submitted PDF still contained values from earlier runs, made before the final run behind Table 1, and we did not regenerate them before submission. In those earlier runs, DBA evaluated candidate plans in only one Doubt world per decision instead of the 19 used in the final configuration (m = 20). This explains why the earlier counts imply DBA scores 5–6 points lower: they lie between the observed-world-only ablation and the m = 5 condition of Table 10 (e.g., DESPITE 72.4% versus 70.0% and 73.6%). The submitted appendix values are not a different aggregation of the main results, and we withdraw them. In the revision, every appendix table that reports the main configuration is regenerated from the same final run as Table 1, and Tables 45–50 report new experiments requested by the reviewers.

Tables whose values changed (all in the revised PDF):

  • Main text: Table 1 (Gaussian-W1 row added, the old Gaussian row relabeled "legacy", and equal-time latency shown as the actual 9.4 s instead of the cap), Table 2 (coverage 91.3%), Table 3 (Gaussian-W1 row), Figure 1 and the case-study values in Section 4.3.
  • Appendix: Tables 8–22, 24–32, 34, 40–44, and Figure 6.
  • New tables: 45 (per-seed counts for both methods), 46 (Gaussian-W1 control), 47 (sampler audit), 48 (component ablations under both samplers), 49 (unmodified native tasks), 50 (paired intervals for the compute controls).

What the corrected evidence shows.

  • Against the strongest baseline, ConceptAgent, the gains are +8.08 [5.71, 10.44] on the safety macro and +7.23 [5.05, 9.41] on the long-horizon macro. All eight Holm-adjusted p-values are below 0.05 (Table 11).
  • Capped at ConceptAgent's planning time, DBA gains +6.74 [5.06, 8.42] over the eight benchmarks (Table 20). Against ConceptAgent at its own native operating point (9.12 calls, 16.16 s), DBA gains +6.72 [4.48, 8.97] on the long-horizon benchmarks (Table 50).
  • With a Gaussian sampler under the same calibration, DBA is still 4.8 to 5.9 points above ConceptAgent on every benchmark, roughly two thirds of the full gain (Tables 1 and 46). Doubt, the filter, and the monitor have positive intervals under this sampler on DESPITE and PARTNR (Table 48).

Claims we narrowed.

  • Headline results are now described as results on assumption-break tasks derived from the benchmarks. Unmodified native tasks are reported separately: the gain is +2.01 [0.66, 3.36] overall, and the long-horizon interval includes zero.
  • TROPE is credited for the representation schema, the operator families, and heavy-tailed edit-size sampling. We no longer claim heavy-tailed sampling as a contribution.
  • All eight primary Holm-adjusted p-values are below 0.05, but not all are below 0.001.
  • Under the time cap, seven of eight intervals exclude zero, and HAZARD's does not.
  • α = 1.6 is a fixed operating point, not an optimum.
  • The red-team advantage depends on the incident corpus and filter, and we report it with that caveat.

Per-seed counts. Table 45 gives the success counts of both methods at every seed, so every reported rate, seed difference, and macro can be recomputed directly from the paper.

We are grateful to all reviewers for feedback that has substantially improved the paper. We hope the corrected results and new controls resolve the main concerns, and we would welcome any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.