- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines whether action-magnitude-based intervention scores reliably measure chain-of-thought (CoT) faithfulness in vision-language-action (VLA) models.
The authors introduce CoT-Faith, a benchmark consisting of 13 intervention families, meaning-preserving controls, size-matched null interventions, and closed-loop evaluations. The experiments cover 11 token-decoded VLA configurations from two model lineages.
The main finding is that magnitude-based faithfulness assessments depend strongly on the choice of control intervention. Two meaning-preserving controls produce opposite comparison signs on 10 of 11 configurations under teacher-forced evaluation. The authors also show that the magnitude score largely reflects whether decoded actions change, without necessarily capturing whether actions follow the modified reasoning semantics.
Closed-loop experiments on two competent DeepThinkVLA policies further demonstrate that modifying or removing CoT does not substantially reduce task success, despite measurable action-level sensitivity.
Critical strengths
Strength 1: Important research question and informative empirical findings. (Significance: 5/5)
The paper addresses an important methodological problem in reasoning-based robotic systems: whether action changes induced by CoT interventions provide reliable evidence of reasoning faithfulness.
The finding that different meaning-preserving controls yield opposite assessments is particularly informative. The additional analysis showing that magnitude scores are strongly associated with action-change frequency provides useful empirical evidence for understanding the limitations of this evaluation approach.
Strength 2: Comprehensive experimental evaluation. (Significance: 4/5)
The authors evaluate multiple intervention families, matched controls, direction-aware diagnostics, and retraining variability across different VLA configurations.
The closed-loop experiments on DeepThinkVLA-SFT and DeepThinkVLA-RL are particularly valuable, as both policies achieve approximately 96% success on LIBERO. The finding that removing CoT does not substantially reduce success, whereas replacing it with another task's CoT significantly reduces success, provides useful behavioral evidence about the limits of action-level faithfulness measurements.
Critical weaknesses
Weakness 1: Limited independent validation of meaning-preserving controls. (Severity: 3/5)
The central finding relies on the semantic validity of paraphrase_null and syntactic_scramble. However, their meaning preservation is primarily validated using a single LLM judge with approximately 40 examples per family. The reported 17.6% disagreement when reversing the presentation order indicates that the judge's assessments are not fully stable.
In particular, word-order changes in syntactic_scramble may affect compositional spatial or action-related meanings even when the original content words are retained. The current validation does not fully establish that these interventions consistently preserve action-relevant semantics.
Suggested Improvement: The authors should conduct an additional human-annotated validation on a representative subset of paraphrase_null and syntactic_scramble examples, explicitly checking whether the intended action, movement direction, and target object remain unchanged. The main magnitude-score comparisons should then be repeated on the independently verified meaning-preserving examples. This would directly test whether the reported opposite verdicts persist after controlling for possible semantic changes.
Weakness 2: Limited analysis of the relationship between action perturbations and closed-loop recovery. (Severity: 2/5)
The closed-loop experiments show that CoT interventions can change individual actions without substantially reducing task success. However, the current results do not establish whether this stability arises because the modified CoT has little influence on task-relevant behavior or because subsequent feedback and replanning compensate for local action deviations.
Although the paper reports trajectory divergence, the analysis does not explicitly characterize how perturbations evolve after successive replanning steps. This limits the interpretation of the relationship between step-level sensitivity and outcome-level reliance.
Suggested Improvement: The authors should provide a time-resolved analysis of the existing paired closed-loop trajectories, measuring how action and end-effector deviations evolve after CoT interventions. In particular, comparing immediate deviations with later trajectory behavior and task completion would help determine whether the observed success stability is consistent with closed-loop compensation. If necessary, a comparison between single-step and persistent interventions could provide additional evidence. The conclusions should distinguish the absence of measurable outcome-level dependence from the absence of an influence on intermediate action selection.
Questions for authors
Q1.
Can the authors validate a representative subset of paraphrase_null and syntactic_scramble edits using independent human annotations of action-relevant semantic equivalence, and recompute the main faithfulness-score comparisons on the verified subset? Do the opposite verdicts observed in Table 2 remain consistent after excluding edits that may alter movement direction, target objects, or intended actions?
Q2.
The closed-loop experiments report substantial action-level sensitivity but little change in task success. Can the authors use the existing paired trajectory data to analyze how action deviations evolve over subsequent replanning steps, particularly for negation and object_swap? Does the observed trajectory behavior support feedback-based compensation, or are the initial action perturbations too small or task-irrelevant to affect success?
Thank you for the constructive review and two concrete questions. We answer them in order; both are addressed by supplementary analyses.
Q1/W1 — Do the opposite verdicts hold once edits that may alter direction, target or action are excluded? Yes; they persist and strengthen. We validated paraphrase_null and syntactic_scramble on the exact scored pairs in three ways.
Q2/W2 — Do perturbations persist, or does the closed loop compensate? Both, depending on the edit. We analysed the existing paired closed-loop trajectories over time. We also added a single-step condition, in which the edit is applied only at the first re-plan and the policy's own CoT is used afterwards.
The revision states your distinction explicitly. The paper shows no measurable outcome-level dependence on the reasoning's content; it does not claim the reasoning has no influence on intermediate action selection.