- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Task-Conditioned Criticality for Verification (TCCV), a framework for embodied task-plan verification and repair. Its central motivation is that a violated condition does not necessarily invalidate a plan: the consequence of a violation depends on the task objective, temporal scope, and whether the plan can recover from the violation.
TCCV distinguishes condition satisfaction, task-conditioned criticality, and evidence uncertainty. It learns condition criticality through matched simulator interventions that measure whether modifying an individual condition changes semantic task success. At inference time, the learned criticality estimates are combined with structured environmental evidence to determine whether to execute, observe, repair, or reject a candidate plan.
The authors evaluate TCCV on ALFRED tasks in AI2-THOR. The framework achieves 99.3% verification accuracy on 282 held-out contrast pairs, outperforming adapted VerifyLLM and VeriGraph baselines. The paper also evaluates condition-level learning through controlled ablations, studies generalization to unseen trajectories, and demonstrates that criticality-guided repair improves over unguided LLM repair in a controlled injected-failure setting.
Critical strengths
- A well-motivated and meaningful formulation of task-conditioned plan verification.
The paper identifies an important distinction between detecting a violated condition and determining whether that violation actually threatens task success.
For example, an initially closed receptacle need not invalidate a plan if the plan opens it before use. Similarly, the relevance of an object's attributes depends on the specific instruction.
Separating condition satisfaction, criticality, and evidence uncertainty provides a conceptually useful framework for reasoning about plan execution.
I also appreciate the explicit handling of temporal scope and the distinction between semantic task success and controller success.
- Execution-grounded supervision and relatively careful experimental methodology.
Rather than relying exclusively on LLM self-assessment, the authors construct matched interventions around audited successful trajectories and derive criticality supervision from execution outcomes.
This is a meaningful methodological choice because it grounds condition importance in observable task consequences.
The paper also includes controlled task–condition ablations, trajectory-disjoint splits, comparisons with existing verification approaches, and robustness analyses involving multiple defects and corrupted environmental evidence.
The authors are appropriately cautious about several findings, particularly the limited statistical evidence for improvement over direct success prediction and the assumptions underlying structured-evidence evaluation.
Critical weaknesses
- The experiments do not establish a sufficiently convincing advantage of learning task-conditioned criticality over simpler, equally informed alternatives.
My primary concern is whether the paper demonstrates that its proposed condition-level learning objective provides a substantial advantage over directly predicting plan success.
The most relevant comparison is E1, where the authors evaluate the learned criticality estimator against a same-input, same-encoder direct-success baseline.
Full TCCV achieves 0.792 balanced accuracy and 0.886 AUROC, compared with 0.767 and 0.870 for direct success prediction. While these results favor TCCV numerically, the improvement in balanced accuracy is only 2.5 percentage points, and the authors acknowledge that the bootstrap confidence interval for the difference includes zero.
Therefore, this experiment does not provide statistically decisive evidence that learning task-conditioned criticality is superior to learning a direct success predictor under matched conditions.
The ablations do demonstrate that the model uses task–condition relationships: removing or shuffling relevant semantic inputs significantly degrades performance. This is useful evidence about the representation, but it does not establish that the proposed learning objective is preferable to a simpler predictor that can also use those relationships.
This distinction is important because the paper's methodological contribution is not merely using task and condition information, but explicitly learning condition-level criticality as an intermediate verification mechanism.
I would find the contribution more convincing if the authors demonstrated a clearer and more consistent benefit over matched-information alternatives, especially on cases where condition relevance cannot be inferred from relatively simple task–condition associations.
- The near-perfect main verification results are obtained in a highly structured, support-constrained setting, limiting their generality.
The headline E2 result of 99.3% verification accuracy is impressive. However, its interpretation requires care.
The evaluation uses structured AI2-THOR evidence and explicitly constructed contrast pairs. Every evaluated task-type–condition cell is represented in training, with at least five training pairs per cell.
These design choices are reasonable for a controlled evaluation, but they also make the problem substantially more constrained than general embodied plan verification.
The authors acknowledge that E2 evaluates supported explicit contrasts and that the near-perfect performance should not be attributed solely to the criticality learner.
I agree with this qualification, but it raises an important question about the paper's main empirical contribution: how much of the observed performance is attributable to task-conditioned criticality learning, rather than the structured evidence, supported task–condition combinations, and highly diagnostic contrast construction?
The E4 generalization experiment provides some additional evidence, but it contains only 25 contrast pairs selected from the existing E2 test split. The examples remain within the same ALFRED/AI2-THOR environment and previously supported task–condition families.
Therefore, E4 is better interpreted as an additional analysis of unseen trajectories within the existing evaluation distribution, rather than an independent demonstration of substantially broader generalization.
I would like to see a more challenging evaluation involving held-out task–condition compositions, greater linguistic variation, or less templated contrasts.
Such experiments would provide stronger evidence that TCCV learns transferable task-conditioned relationships rather than primarily exploiting regularities of the evaluated task families.
- The repair experiments do not fully isolate the contribution of the learned criticality estimator.
The repair results show that TCCV-guided repair can substantially improve over plain LLM repair. On the reported 100-task evaluation, TCCV achieves 98% final completion, compared with 79% for plain LLM repair.
However, the two systems do not receive equivalent diagnostic information.
The TCCV-guided repair prompt includes an explicit task-conditioned violation hint, including the missing or misplaced action and its expected relationship to the task goal. The plain LLM baseline receives only the instruction, candidate plan, and legal action set.
Consequently, the observed improvement demonstrates the usefulness of structured, localized diagnostic information, but does not fully isolate the unique value of learning that information through the proposed criticality estimator.
This concern is reinforced by the VeriGraph repair results. In the evaluated setting, the adapted VeriGraph approach achieves 100% completion and 100% preservation of initially successful plans, compared with TCCV's 98% completion and 96% preservation.
These findings suggest that a sufficiently informative symbolic or scene-graph diagnosis can be at least as effective as learned criticality for the tested repair tasks.
A stronger comparison would include a deterministic or rule-based localized diagnosis baseline that supplies the same type and amount of repair information. This would clarify whether the gains arise specifically from task-conditioned criticality learning or more generally from providing an explicit, actionable repair hint.
Questions for authors
- What is the unique advantage of the learned criticality objective over direct success prediction?
In E1, the improvement over the same-input direct-success baseline is relatively small and not statistically decisive.
Could the authors identify specific categories of task–condition relationships for which TCCV consistently outperforms direct success prediction?
For example, does the learned criticality representation offer measurable advantages under harder semantic contrasts, longer planning horizons, or less templated instructions?
A stronger controlled comparison would materially affect my assessment.
- How much of E2's near-perfect performance depends on the structure of the evaluation protocol?
Since the E2 test set contains only supported task-type–condition combinations and uses structured environmental evidence, could the authors provide a more challenging evaluation that reduces reliance on these constraints?
In particular, holding out selected task–condition compositions or using more varied natural-language instructions would help clarify the generalization capability of the learned estimator.
- Can the repair improvement be separated from the benefit of providing additional diagnostic information?
Could the authors compare TCCV-guided repair with a matched-information symbolic or deterministic diagnostic baseline that supplies equivalent localized repair hints?
This would help determine whether the learned criticality estimator offers advantages beyond the value of an explicit task-specific explanation.
Relatedly, which repair cases can TCCV solve that the adapted VeriGraph baseline cannot?
- How should the framework's reliability be interpreted under imperfect evidence and interacting failures?
The robustness experiments are useful, but they also reveal substantial performance degradation under corrupted structured evidence and reduced complete-defect recall when multiple violations interact.
Could the authors discuss whether these limitations can be addressed by improved evidence uncertainty estimation or multi-condition intervention modeling, and whether the proposed decision mechanism remains reliable under those extensions?
4 comments
Table (a): Generalization Across Test Combinations
| Test combinations | Pairs | TCCV BA | TCCV AUROC | DirectEffect BA |
|---|---|---|---|---|
| Supported | 3507 | 90.60% | 0.975 | 88.65% |
| Held-out | 1284 | 71.81% | 0.667 | 49.26% |
Table (b): Transfer Across Language Variations
| Test combinations | Original BA | Task paraphrase BA | Task + condition paraphrase BA |
|---|---|---|---|
| Supported | 90.60% | 89.29% | 85.65% |
| Held-out | 71.81% | 71.68% | 66.98% |
Table (c): Verification of All Candidate Tasks
| Method | Acc. | BA | Pass | Block | FAR | FRR |
|---|---|---|---|---|---|---|
| TCCV | 75.50% | 80.56% | 91.82% | 69.31% | 30.69% | 8.18% |
These three tables describe the transfer and verification capabilities of TCCV. DirectEffect is a direct-success prediction baseline that uses the same network architecture as TCCV. It predicts task success separately when the queried condition is satisfied and when it is violated. The difference between these probabilities estimates the condition's influence on task success. Unlike TCCV, it learns execution outcomes rather than condition-level criticality labels directly. In Table (a), TCCV achieves 71.81% balanced accuracy on unseen task–condition combinations. This exceeds DirectEffect by 22.55 percentage points and indicates some ability to transfer across combinations. In Table (b), instruction paraphrasing yields balanced accuracies of 89.29% and 71.68%, respectively. These values are close to those obtained with the original wording. When both the task and condition are paraphrased, balanced accuracy falls to 85.65% and 66.98%. Changes in condition wording therefore have a greater impact. In Table (c), TCCV preserves 91.82% of successful plans and rejects 69.31% of failed plans across all 400 candidate task bindings. Its false acceptance rate is 30.69%. These results show both the capabilities and performance differences of condition criticality in combination transfer, language adaptation, and selective verification.
Response to Weakness 3
Thank you for these questions and concerns. We address them below.
(1) TCCV and the LLM baseline do receive different inputs in the repair experiments. The LLM baseline uses only the task, plan, and environmental conditions to decide whether repair or adjustment is needed. TCCV supplies the repairer with the critical-condition violations it detects. This information is central to the improvement in repair accuracy. The input difference is intentional: we aim to compare TCCV-guided LLM repair with LLM repair alone. Under this comparison, the plain LLM baseline should not receive additional error-localization information.
(2) The adapted VeriGraph method in the original manuscript was not a reproduction of the original VeriGraph paper. Our description did not make this sufficiently clear. We have revised the terminology and added a repair implementation based on the original VeriGraph method. The earlier adaptation also received error information identified by TCCV. It was intended to examine the compatibility between TCCV’s core mechanism and VeriGraph’s global causal graph. Although the results were strong, the description obscured the comparison and suggested that TCCV repaired plans less effectively than other baselines.
To better compare repair performance and examine the role of TCCV’s condition criticality, we added further methods. The complete table is shown below, and the manuscript has been updated accordingly.
Table: Repair results on 100 AI2-THOR task contexts derived from official ALFRED high-level plans, including 50 initially failed and 50 initially successful plans.
| Policy | Final Comp. ↑ | Repair Succ. ↑ | Success Pres. ↑ | Avg. Calls ↓ |
|---|---|---|---|---|
| No Repair | 0.500 | 0.000 | 1.000 | 0.000 |
| LLM Repair | 0.740 | 0.680 | 0.800 | 1.210 |
| VeriGraph (native-algorithm) | 0.800 | 0.760 | 0.840 | 6.830 |
| VeriGraph + TCCV diagnosis† | 0.880 | 0.860 | 0.900 | 5.080 |
| TCCV | 0.960 | 0.940 | 0.980 | 0.920 |
Notes. Comp., Succ., Pres., and Calls denote final completion, repair success, success preservation, and average repair-stage LLM calls per context, respectively. † This row uses the earlier native-algorithm VeriGraph adaptation with TCCV-generated diagnostic feedback.
The table above compares five repair policies on 50 initially failed and 50 initially successful plans. TCCV achieves 96% final completion, compared with 74% for LLM Repair, 80% for the native-algorithm VeriGraph adaptation, and 88% for VeriGraph with TCCV diagnosis. TCCV repairs 47 of the 50 failed plans and preserves 49 of the 50 successful plans. Its average repair-stage cost is 0.92 LLM calls per context, including contexts that require no repair.
The methods differ in the feedback supplied to repair. LLM Repair receives the task, candidate plan, action interface, and shared goal and object information. VeriGraph uses current and desired scene graphs with iterative graph-checking feedback. VeriGraph + TCCV adds the earliest critical violation and its associated plan step. TCCV uses task-conditioned risk to decide whether to repair and supplies the selected obligation and current evidence to the repairer. In the new repair experiments, we adopt a unified evaluation criterion. Each repaired plan is executed in AI2-THOR using the evaluation controller. Repair success requires the final semantic goal to be satisfied. Among all failed plans subjected to repair, one was classified as a repair failure because all three repair attempts timed out. This is consistent with its failure status before repair.
Adding TCCV diagnosis to VeriGraph increases repair success from 76% to 86%, corresponding to five additional repaired plans. Success preservation rises from 84% to 90%, and final completion improves by eight percentage points. Average repair-stage calls decrease from 6.83 to 5.08. These gains show that localized critical-condition feedback can improve a graph-based repair pipeline. Compared with LLM Repair, TCCV raises repair success from 68% to 94% and success preservation from 80% to 98%, while reducing average calls from 1.21 to 0.92. Thus, the TCCV pipeline both recovers more failed plans and protects more valid plans.
Responses to the Additional Questions:
Response to Question 1.
We acknowledge that E1 does not establish a statistically significant advantage of the criticality objective over Direct Success with the same inputs. The original manuscript already states this limitation. E1 mainly shows that the model uses task–condition interactions instead of relying only on a condition’s static importance. TCCV directly learns condition-level success flips and uses these estimates for local risk identification and selective repair. We added experiments to further distinguish this inductive bias from direct success prediction and examine generalization and performance stability. The experiments and analyses are described in detail in our response to Weakness 2.
Response to Question 2.
We agree that E2’s high accuracy must be interpreted in the context of its structured evaluation protocol. During training, we use condition flips to assess criticality. The test set therefore also includes combinations involving conditions whose criticality has been established. For combinations that have never been learned, we would expect TCCV to perform comparably to a conventional PDDL-based approach.
Section 5.3 of the original manuscript explicitly positions E2 as a test of the complete verification pipeline within supported conditions. It is not presented as evidence of open-ended generalization. Appendix B also includes an independent composition test involving six held-out task–condition pairs. E4 reports results from frozen models on valid-unseen trajectories. Thus, our evaluation is not limited to task contexts identical to those seen during training. We also keep condition combinations fixed while changing the context and wording. The additional experiments are described in detail in our response to Weakness 2.
Response to Question 3.
Thank you for this suggestion. We added a “VeriGraph + TCCV diagnosis” comparison. The same VeriGraph planner receives the critical violations localized by TCCV. It does not receive ground-truth error hints or reference repair plans. In this algorithm adaptation, both repair success and preservation of valid plans improve.
In the following two cases, TCCV-guided repair passed high-level semantic validation, whereas the adapted VeriGraph method did not:
-
Case One: Missing final placement action. The task requires disposing of a hand towel. The original plan only picks up the towel and navigates to the garbage can. The TCCV-guided repairer adds an action to place the towel in the garbage can, satisfying the final goal. The VeriGraph output still fails to meet the placement requirement.
-
Case Two: Incorrect order of a processing action. The original plan schedules cleaning the kettle after its final placement. The TCCV-guided repairer moves cleaning to an earlier stage, while the agent still holds the kettle and before placement. The VeriGraph output fails to satisfy the required processing state.
These cases illustrate how task-relevant local diagnosis can help with missing actions and ordering errors when the objects are clearly identified and the required edits are small. They do not imply that VeriGraph is inherently unable to solve these cases, nor do they establish overall superiority for TCCV. We also retain comparisons using a shared repair backend and constant criticality. The constant-criticality control currently produces the same outputs as the retrained TCCV model. We therefore have not established an independent advantage of the learned scores beyond explicit diagnostic information itself.
Response to Question 4:
Thank you for raising reliability under multiple defects and imperfect evidence. We clarify that the decrease reported in Appendix Table 6 concerns all-defect recall. Any-defect recall reaches 100% in both the two-defect and three-defect settings. Detecting the presence of failure and fully localizing every defect therefore have different reliability levels in the current framework.
This pattern is consistent with TCCV’s bottleneck risk aggregation and sequential repair design. The system prioritizes the highest-risk obligation and recomputes subsequent risk after repair. However, the current experiments do not establish that all interacting defects can be repaired reliably.
For evidence noise, Section 4.3 explicitly supports unknown evidence and the OBSERVE decision. However, Appendix B.2 also shows that severely incorrect evidence substantially increases FAR. We see the following directions for improvement:
We could calibrate evidence confidence, repeat observations, and use joint intervention supervision for interacting conditions. Any new mechanism must be reassessed for FAR under matched coverage and execution budgets. We cannot assume that the current thresholds and decision guarantees will remain valid. Errors may also influence one another. A later error may result from an earlier one and disappear once the earlier error is corrected. Other errors may be independent. We plan to study similarities between errors and their joint effects across tasks. We believe this is a useful direction for future work.
Finally, thank you for taking the time to provide constructive feedback and recognize our work. We hope these responses clarify our approach and help you reassess its value. Best regards.
Thank you for your careful review and constructive suggestions. We address your comments individually below.
Response to Weakness 1
In the original submission, E1 was designed to examine the importance of task–condition criticality and the effect of disrupting semantic order. As you noted, the results did not show a statistically significant advantage for TCCV. We understand the concern this raises. Although TCCV performed well in E2, we should also examine why its E1 performance was close to that of Direct Success. We hope the following explanation clarifies our work.
We attribute the similar performance of TCCV and Direct Success in the original manuscript to two factors:
Both methods were trained with a fixed plan. Their inputs contained task semantics and condition criticality information. This restricted the information available to the models and reduced the performance difference.
The dataset contained many task-flip combinations, with only one action changed at a time. This resembles the paired construction used to identify condition criticality. Direct Success may therefore also learn information about condition criticality, leading to similar performance.
Based on this analysis, we added new comparisons with explicit plan information. The new dataset includes the states of all conditions relevant to the task and plan, rather than changing only one environmental condition. Condition states are generated randomly. This setting reflects the information available in realistic inputs. For TCCV, we select condition-flip pairs to learn criticality. Its training data include these pairs and the remaining records. We compare two Direct Success variants. One learns directly from the original data. The other also receives criticality information learned by TCCV. We additionally compare TCCV under plan shuffling. The detailed results are shown below and have been added after the original Table 2 in the revised manuscript.
Table 1: Comparison under the same downstream evaluation protocol.
Table 1 provides a complementary evaluation with explicit plan inputs. Direct predicts task success using 64 downstream outcome labels. Direct (+ criticality features) additionally uses predictions from an independently pretrained, frozen criticality estimator. Standalone TCCV combines frozen semantic criticality with current condition violations. Adding criticality features improves Direct's balanced accuracy from 82.85% to 88.26% and reduces FAR from 16.22% to 6.13%. Standalone TCCV achieves 98.71% balanced accuracy and 0.9962 AUROC.
The performance difference between TCCV and Direct is larger in this supplemental evaluation than in E1. One explanation is that the two settings capture different aspects of condition criticality. E1 isolates task–condition alignment under a fixed plan, while the supplemental evaluation includes explicit plan representations. Direct-success supervision may also capture some condition-relevant information from contrastive training examples. However, the datasets, model configurations, and supervision budgets differ. The results therefore do not isolate the effect of plan encoding or establish a same-budget advantage of criticality learning.
Response to Weakness 2
TCCV performs strongly in the original E2 comparison. Its performance benefits from condition criticality learned from the training data. Combinations not covered by that learning were outside the scope of the original experiment. We also expanded the generalization evaluation to include changes in wording and conditions. These experiments supplement the generalization analysis in E4 and have been added to the revised manuscript.
The new experiments include unseen task–plan–condition combinations and greater variation in input language. The results are also presented in Figure 3 of the revised manuscript. They indicate that the model transfers to some unseen combinations.