- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces DOUBTBEFOREACT, an inference-time framework for improving the safety of embodied LLM planning by explicitly examining uncertain assumptions before execution.
The framework extracts task-relevant assumptions from the current scene and uses typed representation-editing operators in two modes. Propose generates alternative candidate plans, while Doubt constructs plausible counterfactual worlds in which the original assumptions may fail. Candidate plans are evaluated across these worlds, and a risk-sensitive selection rule based on Conditional Value at Risk (CVaR) chooses the plan with the best lower-tail performance. A runtime monitor subsequently checks whether the assumptions supporting the selected plan remain valid.
The authors evaluate the framework on five embodied safety benchmarks and three long-horizon planning benchmarks, comparing against thirteen baselines. The paper also includes ablation studies, held-out assumption classes, independently generated test worlds, computational budget controls, and a separate red-teaming evaluation.
Critical strengths
- Meaningful safety problem formulation.
The paper addresses an important limitation of embodied LLM planning: a plan that appears valid under the currently observed state may become unsafe when its underlying assumptions fail. Explicitly identifying these assumptions and examining possible violations before execution is a relevant direction for improving embodied-agent safety.
- Broad experimental coverage and useful methodological controls.
The evaluation covers multiple embodied safety and long-horizon benchmarks and includes a substantial collection of relevant baselines. The authors also make efforts to evaluate generalization beyond scenarios generated by their own method through held-out assumption classes and independently authored test worlds.
The inclusion of component ablations, equal-wall-clock comparisons, and additional compute-budget controls is valuable. These experiments make the proposed framework more thoroughly evaluated than a simple comparison against unmodified LLM planners.
However, the serious numerical inconsistencies discussed below substantially undermine confidence in the reported empirical conclusions.
Critical weaknesses
- Serious numerical inconsistencies across the main results and appendix undermine the reliability of the central experimental claims.
My primary concern is that several reported experimental results cannot be reconciled using the definitions and numerical counts provided in the paper.
First, the main results in Table 1 are inconsistent with the full-method success counts reported in Appendix Table 13.
For example:
- DESPITE: Table 1 reports 78.1% success, whereas Appendix Table 13 reports 3,779 successes out of 5,220 evaluations, corresponding to approximately 72.39%.
- EMBODYGUARD: Table 1 reports 79.8%, whereas 2,562/3,420 corresponds to approximately 74.91%.
- SafeAgentBench: Table 1 reports 69.8%, whereas 2,378/3,750 corresponds to approximately 63.41%.
- IS-Bench: Table 1 reports 63.2%, whereas 458/805 corresponds to approximately 56.89%.
- HAZARD: Table 1 reports 66.5%, whereas 672/1,110 corresponds to approximately 60.54%.
These differences are substantial and cannot be explained by ordinary rounding.
Second, Appendix Table 13 contains an additional inconsistency between the reported per-seed success-count differences and the corresponding percentage-point improvements.
For DESPITE, the five reported success-count differences are [46, 71, 51, 68, 56], totaling 292 additional successes. With the reported total evaluation denominator of 5,220, this corresponds to approximately 5.59 percentage points, rather than the reported mean improvement of 8.0 percentage points.
Since Table 13 explicitly defines the per-seed differences using success counts and the number of paired task IDs, these quantities should be reconcilable under a clearly specified aggregation procedure.
Third, Appendix Table 14 reports a safety macro-average of 65.6% and a long-horizon macro-average of 63.8% for the full method. However, directly averaging the corresponding full-method results in Table 1 yields approximately 71.48% and 69.50%, respectively. Both tables describe equal-weight macro-averages over the same five safety and three long-horizon benchmarks.
These inconsistencies affect the primary results rather than peripheral analyses. In particular, the claimed improvements, statistical confidence intervals, and conclusions regarding performance across multiple benchmarks depend on the correctness of these underlying measurements.
I cannot determine from the manuscript whether the discrepancies arise from different experimental populations, undocumented configurations, aggregation errors, or inconsistent result versions. Nevertheless, the current paper does not provide a sufficient explanation.
Until the authors supply a consistent mapping between the reported counts, denominators, experimental conditions, and summary statistics, I cannot regard the headline experimental improvements as reliably established.
This issue is particularly consequential because the paper's main contribution is supported primarily by empirical performance improvements across multiple benchmarks.
- Limited independent methodological novelty and unresolved concerns about the interpretation of computational comparisons.
The proposed framework combines several established techniques, including structured assumption representation, counterfactual scenario generation, robust planning, CVaR-based selection, and runtime monitoring.
The authors acknowledge that CVaR and scenario-based robust optimization are established methods, while the typed editing operators are adapted from the prior TROPE framework. The main methodological contribution therefore appears to lie in integrating these components into an embodied planning pipeline rather than introducing a fundamentally new risk-sensitive planning or uncertainty-reasoning mechanism.
Such integration could still constitute a meaningful contribution if its independent advantages were demonstrated convincingly. However, the numerical inconsistencies in the main results currently prevent a reliable assessment of the claimed gains.
I also have concerns about the interpretation of the matched-compute evaluation.
The headline protocol allocates five LLM calls and 80 execution evaluations to every method. However, Appendix Table 18 distinguishes decision-visible work from accounting-only operations that cannot affect the returned decision.
Under this accounting, DOUBTBEFOREACT uses all 80 evaluations for decision-making, whereas ReAct, Tree of Thoughts, SafeMem, and RoboSafe use 14, 38, 20, and 22 decision-visible evaluations, respectively.
The authors explicitly disclose this distinction and provide additional controls, including matched-native-use, equal-wall-clock, and reduced-execution comparisons. These efforts are appreciated, and I do not consider the nominal-budget issue alone sufficient to invalidate the study.
Nevertheless, equal nominal resource consumption should not be interpreted as equal effective decision-making computation. The central performance claims should be supported by clearly identified, consistent results under the relevant effective-budget conditions.
Finally, the approach relies on generating plausible counterfactual worlds and evaluating plans using symbolic or simulator-backed execution. Although this is appropriate for the studied benchmarks, the ability of the resulting planner to handle omitted assumptions, inaccurate transition models, or physically invalid counterfactual scenarios remains insufficiently established.
Overall, the proposed combination is potentially useful, but its independent methodological contribution and empirical advantages are not convincingly demonstrated in the current submission, particularly given the unresolved inconsistencies in the primary results.
Questions for authors
Q1. Can the authors explain and reconcile the numerical discrepancies between Table 1 and Appendix Table 13?
For example, Table 1 reports 78.1% success on DESPITE, while Table 13 reports 3,779/5,220 full-method successes, corresponding to approximately 72.39%.
Similar discrepancies occur across all five safety benchmarks.
What are the exact evaluation populations, denominators, model configurations, and aggregation procedures underlying these two tables? Do they correspond to different experiments or versions of the evaluation?
Please provide the underlying per-task, per-seed success records and the aggregation procedure used to generate the reported results.
Q2. Why are the reported success-count differences and macro-averages inconsistent with their stated definitions?
Appendix Table 13 reports five DESPITE success-count differences totaling 292, which corresponds to approximately 5.59 percentage points over 5,220 evaluations, rather than the reported 8.0-point mean improvement.
Similarly, Appendix Table 14 reports full-method safety and long-horizon macro-averages of 65.6% and 63.8%, while the corresponding results in Table 1 imply approximately 71.48% and 69.50%.
Can the authors clarify whether these differences arise from undocumented evaluation subsets, alternative aggregation procedures, or errors in the reported tables?
If the primary tables require correction, how would this affect the reported improvements, paired confidence intervals, statistical significance tests, and conclusions?
Q3. What is the independent advantage of the proposed assumption-level counterfactual editing mechanism under comparable effective computation?
The framework combines established scenario-based robust planning and CVaR optimization with typed editing operators adapted from prior work.
Can the authors clarify which component provides the principal methodological advancement beyond conventional uncertainty-aware planning?
In addition, given the distinction between nominal and decision-visible computation in Appendix Table 18, can the authors identify the results that most directly establish superiority under comparable effective planning resources, with paired uncertainty estimates?
Answers to these questions, particularly Q1 and Q2, would materially affect my assessment of the submission.
4 comments
(2/2)
Q3. Independent contribution and comparable effective resources.
- We frame the contribution as an embodied integration, not as a new CVaR objective or a new heavy-tailed distribution. TROPE supplies the representation schema, the operator families, and edit-size sampling. DBA uses scene assumptions as an explicit interface that links candidate generation, counterfactual world construction, plausibility checks, plan selection, and runtime monitoring.
- With Gaussian-W1 sampling, DBA remains 4.8 to 5.9 points above ConceptAgent on every benchmark (Table 1), roughly two thirds of the full gain.
- Table 48 removes each component under Gaussian-W1. Doubt contributes +5.31 [3.50, 7.12] on DESPITE, +4.00 [2.07, 5.93] on PARTNR, and +4.20 [0.64, 7.76] on EB-ALFRED. The filter and the monitor also have positive intervals on DESPITE and PARTNR, while on EB-ALFRED their intervals include zero.
We agree that the nominal limit of 5 calls and 80 evaluations does not equalize decision-making work. The revised Table 18 reports the calls and evaluations that each method actually completes in the final run. The decision-visible counts you quote (14/38/20/22) came from the superseded appendix. Three controls address different resource questions, each with paired intervals:
- Under the time cap (Table 20), DBA is limited to ConceptAgent's per-benchmark planning time (mean cap 9.98 s, actual DBA mean 9.41 s). The gains are +7.00 [4.65, 9.34] on the safety macro, +6.31 [4.11, 8.52] on the long-horizon macro, and +6.74 [5.06, 8.42] over all eight benchmarks. Seven of the eight benchmark intervals exclude zero, and HAZARD's is [−0.56, 15.34]. This is the most direct comparison under equal measured resources.
- At its native operating point (Tables 22 and 50), ConceptAgent uses 9.12 calls and 16.16 s, versus 5 calls and 11.23 s for DBA. DBA's gains are +8.30 [5.74, 10.86] on PARTNR, +6.20 [2.77, 9.63] on EAI, and +5.67 [0.41, 10.92] on EB-ALFRED, with a macro of +6.72 [4.48, 8.97]. This removes the five-call restriction on tree search.
- With reduced evaluations (Tables 28 and 50), m = 5, DBA uses 20 evaluations, against RoboSafe's 14.14 completed evaluations. The gains are +6.80 [4.10, 9.50] on DESPITE, +5.10 [2.23, 7.97] on PARTNR, and +6.33 [1.21, 11.45] on EB-ALFRED, with a macro of +6.08 [3.93, 8.23]. The number of evaluations is closer but still not equal.
These controls do not establish identical FLOPs or token use across methods, and we do not claim that they do.
Omitted assumptions, transition errors, and invalid worlds.
- DBA can only stress-test the assumptions it extracts, using the execution interface it is given. An omitted premise cannot be recovered by changing the sampler, and the monitor checks only represented assumptions.
- Schema validation, plausibility filtering, held-out assumption classes, and independently authored test worlds address specific failure modes. They do not prove physical validity or correct transition models.
- The Limitations section now states this, and we do not infer real-robot safety from benchmark results.
Thank you again for the constructive review. You noted that answers to Q1 and Q2 would materially affect your assessment. We hope the corrected counts, intervals, and controls above provide them, and we would be grateful if you would consider revisiting your score. We are happy to clarify anything further.
Assessment of the Rebuttal and Revised Manuscript
Thank you for the detailed rebuttal and substantial revisions. I appreciate the authors' transparency in acknowledging the numerical inconsistencies and providing corrected results with additional experimental controls.
The revised per-seed counts are consistent with the main results, and the new paired confidence intervals, Gaussian-W1 comparisons, and computational controls considerably strengthen the empirical evidence. I therefore consider my original concern regarding numerical reliability substantially addressed.
Nevertheless, two concerns remain.
1. Methodological novelty and generalization. DBA primarily integrates existing representation-editing techniques, counterfactual evaluation, CVaR-based planning, and runtime monitoring. While the integration is meaningful, its independent methodological novelty remains limited. Moreover, the reported gain decreases from +7.76 percentage points on assumption-break tasks to +2.01 points on unmodified native tasks, with the native long-horizon confidence interval including zero. This limits the generality of the conclusions.
2. Statistical clustering. Some benchmark-native tasks generate multiple derived assumption-break instances. Could the authors clarify whether the paired bootstrap clusters by the original native task ID or by individual derived instances? If the latter, a native-task-level sensitivity analysis would help assess the impact of within-task dependence.
Overall, the rebuttal substantially improves the paper and resolves my most serious original concern. However, I remain unconvinced that the methodological novelty and generalization evidence are sufficient for acceptance.
I have therefore increased my rating from 1 (Clear Rejection) to 2 (Weak Rejection).
Thank you again for the constructive response.
Thank you for the careful reassessment and for raising your score. We answer the two remaining points.
Statistical clustering. The bootstrap in Table 11 resampled derived assumption-break instances, with all five seeds of an instance kept together. Following your suggestion, we reran it clustered by native task ID, keeping all derived instances of a native task and all five seeds together:
| Effect | Native-task-clustered 95% CI |
|---|---|
| Safety macro | +8.08 [5.49, 10.67] |
| Long-horizon macro | +7.23 [4.99, 9.48] |
| All-8 macro | +7.76 [5.94, 9.58] |
| HAZARD | +8.47 [−1.17, 18.11] |
The macro conclusions are unchanged. The HAZARD interval now includes zero, so we no longer claim a significant per-benchmark effect on HAZARD. The next version will report the native-task-clustered intervals as the primary ones and revise the per-benchmark claims accordingly.
Novelty and generalization. We agree that the contribution is an integration, and the revised paper states it that way. On generality, DBA is designed for the case where an assumption the plan relies on breaks. In unmodified native tasks no break is injected, so the counterfactual check has less to act on, and a smaller gain is the expected outcome. We report it as measured: +2.01 [0.66, 3.36] overall, with positive intervals on DESPITE and PARTNR and an inconclusive long-horizon macro, and we do not extend the assumption-break gains to native tasks.
Thank you again for the constructive engagement.
(1/2)
Thank you for separating the numerical, methodological, and computational concerns. We agree that the appendix discrepancies were substantive. The revision replaces the inconsistent appendix summaries, supplies seed-level counts for both methods, and adds paired intervals for each resource control.
Q1. Evaluation populations, versions, and aggregation. The submitted manuscript combined the main results of the final run with appendix tables and comparator labels from an earlier result version (general response). The old counts are not a different rounding or weighting of the main results, and we withdraw them.
Q2. Seed differences, macros, intervals, and significance.
Table 11 reports the updated primary inference: