acceptodds
Under review as a conference paper at ICLR 2027

CoT-Faith: Magnitude Scores Do Not Measure Faithfulness in Manipulation VLAs

Abstract

Vision-language-action models (VLAs) that emit chain-of-thought (CoT) are proposed as a route to interpretable robotics, on the premise that the emitted reasoning drives the action. The standard test scores an edit to the reasoning by how often it moves the action. We introduce CoT-Faith, a paired-intervention benchmark with 13 edit families, size-matched nulls and closed-loop evaluation, and apply it to 11 token-decoded VLA configurations from two model lineages. We show that the test’s verdict depends on which meaning-preserving edit serves as its floor: under teacher-forced scoring, two judge-validated floors give opposite verdicts on 10 of 11 configurations, including the two competent policies. The score mostly records whether the action changes at all, which depends on what an edit touches rather than what it means. Nulls matched to each edit in size and tag restore a consistent sign, with direction flips beating their null on 11 of 11, but a control fine-tuned without a CoT target passes them too. Requiring the action to point the implied direction moves the top-ranked model from first of eight to seventh. On two competent LIBERO policies the action is sensitive to the reasoning at each step, yet editing or even emptying it leaves closed-loop success unchanged: sensitivity is not reliance. Edit-based scores should be read against matched nulls, a direction-aware score and closed-loop success before they license a claim.

Primary area:
interpretability and explainable AI
Keywords:
chain-of-thought faithfulness, vision-language-action models, interpretability evaluation, perturbation-based explainability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4fLzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether action-magnitude-based intervention scores reliably measure chain-of-thought (CoT) faithfulness in vision-language-action (VLA) models.

The authors introduce CoT-Faith, a benchmark consisting of 13 intervention families, meaning-preserving controls, size-matched null interventions, and closed-loop evaluations. The experiments cover 11 token-decoded VLA configurations from two model lineages.

The main finding is that magnitude-based faithfulness assessments depend strongly on the choice of control intervention. Two meaning-preserving controls produce opposite comparison signs on 10 of 11 configurations under teacher-forced evaluation. The authors also show that the magnitude score largely reflects whether decoded actions change, without necessarily capturing whether actions follow the modified reasoning semantics.

Closed-loop experiments on two competent DeepThinkVLA policies further demonstrate that modifying or removing CoT does not substantially reduce task success, despite measurable action-level sensitivity.

Critical strengths

Strength 1: Important research question and informative empirical findings. (Significance: 5/5)

The paper addresses an important methodological problem in reasoning-based robotic systems: whether action changes induced by CoT interventions provide reliable evidence of reasoning faithfulness.

The finding that different meaning-preserving controls yield opposite assessments is particularly informative. The additional analysis showing that magnitude scores are strongly associated with action-change frequency provides useful empirical evidence for understanding the limitations of this evaluation approach.

Strength 2: Comprehensive experimental evaluation. (Significance: 4/5)

The authors evaluate multiple intervention families, matched controls, direction-aware diagnostics, and retraining variability across different VLA configurations.

The closed-loop experiments on DeepThinkVLA-SFT and DeepThinkVLA-RL are particularly valuable, as both policies achieve approximately 96% success on LIBERO. The finding that removing CoT does not substantially reduce success, whereas replacing it with another task's CoT significantly reduces success, provides useful behavioral evidence about the limits of action-level faithfulness measurements.

Critical weaknesses

Weakness 1: Limited independent validation of meaning-preserving controls. (Severity: 3/5)

The central finding relies on the semantic validity of paraphrase_null and syntactic_scramble. However, their meaning preservation is primarily validated using a single LLM judge with approximately 40 examples per family. The reported 17.6% disagreement when reversing the presentation order indicates that the judge's assessments are not fully stable.

In particular, word-order changes in syntactic_scramble may affect compositional spatial or action-related meanings even when the original content words are retained. The current validation does not fully establish that these interventions consistently preserve action-relevant semantics.

Suggested Improvement: The authors should conduct an additional human-annotated validation on a representative subset of paraphrase_null and syntactic_scramble examples, explicitly checking whether the intended action, movement direction, and target object remain unchanged. The main magnitude-score comparisons should then be repeated on the independently verified meaning-preserving examples. This would directly test whether the reported opposite verdicts persist after controlling for possible semantic changes.

Weakness 2: Limited analysis of the relationship between action perturbations and closed-loop recovery. (Severity: 2/5)

The closed-loop experiments show that CoT interventions can change individual actions without substantially reducing task success. However, the current results do not establish whether this stability arises because the modified CoT has little influence on task-relevant behavior or because subsequent feedback and replanning compensate for local action deviations.

Although the paper reports trajectory divergence, the analysis does not explicitly characterize how perturbations evolve after successive replanning steps. This limits the interpretation of the relationship between step-level sensitivity and outcome-level reliance.

Suggested Improvement: The authors should provide a time-resolved analysis of the existing paired closed-loop trajectories, measuring how action and end-effector deviations evolve after CoT interventions. In particular, comparing immediate deviations with later trajectory behavior and task completion would help determine whether the observed success stability is consistent with closed-loop compensation. If necessary, a comparison between single-step and persistent interventions could provide additional evidence. The conclusions should distinguish the absence of measurable outcome-level dependence from the absence of an influence on intermediate action selection.

Questions for authors

Q1.

Can the authors validate a representative subset of paraphrase_null and syntactic_scramble edits using independent human annotations of action-relevant semantic equivalence, and recompute the main faithfulness-score comparisons on the verified subset? Do the opposite verdicts observed in Table 2 remain consistent after excluding edits that may alter movement direction, target objects, or intended actions?

Q2.

The closed-loop experiments report substantial action-level sensitivity but little change in task success. Can the authors use the existing paired trajectory data to analyze how action deviations evolve over subsequent replanning steps, particularly for negation and object_swap? Does the observed trajectory behavior support feedback-based compensation, or are the initial action perturbations too small or task-irrelevant to affect success?

1 comment

Thank you for the constructive review and two concrete questions. We answer them in order; both are addressed by supplementary analyses.

Q1/W1 — Do the opposite verdicts hold once edits that may alter direction, target or action are excluded? Yes; they persist and strengthen. We validated paraphrase_null and syntactic_scramble on the exact scored pairs in three ways.

  • Judge panel. Five judges from four model families judged every exact scored pair, not a 40-example sample.
    • Each first passed identity, negative-control and order gates.
    • A pair counts as meaning-preserving only if a judge says so in both presentation orders (cross-order agreement 0.855–0.912), which addresses the order instability you note.
    • The panel's majority rates paraphrase_null meaning-preserving on 1.00 and syntactic_scramble on 0.94, against 0.00–0.04 for meaning-changing edits.
  • Edits that may alter direction. Targeting the compositional word-order issue you raise, we also exclude the 18% of syntactic_scramble edits that reorder the directions of a compound MOVE ("move back and left" → "left and back move"): in the LIBERO annotations the first-named direction is the larger one (130/130 steps; both orders are common in the corpus), so the reorder can change the intended movement.
  • Human annotation. A blind human annotation of exactly these properties (action, direction, target object, gripper state) on a random subset, with the comparison recomputed on the pairs it verifies, gives the same signs on all 8 ECoT-lineage configurations (attention checks 16/16 correct).
  • Result. On the validated pairs the opposite verdicts persist and strengthen: the semantic mean is below the paraphrase floor on 11/11 configurations (8 Holm-significant) and above the scramble floor on 11/11 (7 Holm-significant). With the reorders kept, it is still below the paraphrase floor on 11/11 and above the scramble floor on 8/11, none significantly below it, and the two competent DeepThinkVLA policies stay significant against both floors.

Q2/W2 — Do perturbations persist, or does the closed loop compensate? Both, depending on the edit. We analysed the existing paired closed-loop trajectories over time. We also added a single-step condition, in which the edit is applied only at the first re-plan and the policy's own CoT is used afterwards.

  • Measure: d is the end-effector distance from the clean arm's path, in metres.
  • Noise floor: a re-tokenized but unedited CoT.
  • Data: DeepThinkVLA-RL, 400 paired episodes per arm (SFT is nearly identical).
armd@50d@200peakΔSR persistentΔSR single-step
noise floor.000.016.019——
paraphrase_null.010.048.051−0.003—
negation.012.053.054+0.005+0.018 (n.s.)
object_swap.011.047.050+0.022+0.003 (n.s.)
empty CoT.016.058.063+0.000−0.003 (n.s.)
cross-task CoT.022.081.090−0.158−0.045 (p = .008)
  • Within-task edits (incl. negation and object_swap) are small and content-independent.
    • Semantic and meaning-preserving edits perturb the path by the same amount at every horizon, about 5 cm at peak.
    • A single perturbation already produces that full deviation, and repeating the edit at every re-plan does not add to it.
    • The paths partly reconverge (final ≈ 0.65× peak), and success stays within 0.022 of the clean arm in both conditions.
  • The loop compensates for disruptive reasoning. A one-off cross-task CoT costs 4.5 points; a persistent one costs 15.8.
  • Why content matters so little. We crossed {mirror the image} × {flip the instruction} × {flip the CoT}. On these policies, a scene-consistent direction edit in the CoT changes the action no more than mirroring the image alone (lateral reversal 0.19–0.21 vs 0.16–0.19), and a CoT-only flip of their own reasoning reverses on 0%. Only the image moves their direction.

The revision states your distinction explicitly. The paper shows no measurable outcome-level dependence on the reasoning's content; it does not claim the reasoning has no influence on intermediate action selection.

Reviewer_x6mgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a genuinely hard methodological problem, namely how to evaluate whether a VLA's CoT is faithful to what actually drives its actions.

Critical strengths

The benchmark is impressively thorough, with 13 base edit families, two size-matched nulls, 11 model configurations, and both open-loop and closed-loop evaluations. The central finding is striking. Two judge-validated, supposedly meaning-preserving controls can return opposite faithfulness verdicts for the same model, and the authors trace much of this discrepancy to a simple factor, namely whether the action changes at all. That alone is strong evidence against magnitude-based faithfulness scores. I also appreciate that the authors went beyond open-loop action deltas. In the closed-loop evaluation on policies that can actually complete the tasks, editing or even removing the CoT barely hurts success. Distinguishing action sensitivity from genuine reliance on reasoning makes the conclusions far more useful in practice.

Critical weaknesses

My main concern is that the meaning-preserving controls haven't been independently validated. The headline claim — that the two controls give opposite verdicts — only holds if the controls really do preserve meaning. Right now only Qwen2.5-7B-Instruct passes the authors' judge validation, and that judge looks fragile: reversing the presentation order flips 17.6% of its judgments. The attempted Mistral-7B-Instruct-v0.2 judge is worse, labeling 94.3% of the meaning-changing negative controls as meaning-preserving, so it cannot corroborate anything. There is also a mismatch in what was checked: the roughly 40 inspected pairs per family do not line up record-by-record with the exact texts used for action scoring. I'd like to see the validation redone on the actual scored pairs, with judges from different model families and scales that first clear the same quality-control gates, together with an analysis of whether the disputed examples are the ones driving the opposite-verdict conclusion. If the conclusion survives this, it becomes much stronger; if not, the central result needs reframing. A second, more bounded issue: the matched-control comparison covers only direction_flip and negation among the seven semantic edit families. Recommending that every semantic edit be compared against a size- and tag-matched null is sound, but the paper can only demonstrate this for two families, so the corresponding results should not be read as applying to the rest. The authors should explain why these two were chosen, and any claim meant to cover all seven families needs additional matched controls to back it up.

Questions for authors

When the exact scored pairs are re-judged by models from other families that pass the negative-control gates, do the two controls still disagree? If the disputed pairs flip, which way does the aggregate verdict go? What makes direction_flip and negation special? What are the main technical obstacles to constructing approximately matched nulls for the remaining five families?

1 comment

Thank you for the thorough review and for specifying exactly the checks that would strengthen the central result. We ran them all, and the central result survives each one and becomes stronger.

Q1 — Re-judged by other families, do the two controls still disagree? Yes.

  • Exact pairs. We rebuilt the exact (original, edited) pair for every scored record; all 66,229 match the released edit metadata.
  • Judges. A panel of five judges from four model families, spanning 7B to 104B, judged every pair: Qwen2.5-7B, Qwen2.5-32B, gemma-2-27B, Mistral-Small-24B and Command-R+.
    • Each judge first passed identity, negative-control and order gates. Judges that accept many meaning-changing edits fail the negative-control gate and are excluded.
    • On the order flips you note: a pair counts as preserved only if a judge says so in both orders, so order instability can only make a control harder to validate. Each judge agrees across orders on 0.855–0.912 of exact pairs (Qwen2.5-7B: 0.874).
  • Validation. Panel-majority meaning-preserving rates on the exact scored texts:
rate
paraphrase_null1.00
syntactic_scramble0.94
direction-matched null0.985
negation-matched null0.993
direction_flip / negation0.00
other meaning-changing families0.00–0.04
  • Verdict on panel-accepted pairs. The two controls still disagree: the semantic mean is below the paraphrase floor on 11/11 configurations (7 Holm-significant) and above the scramble floor on 8/11, none significantly below it. On the two competent policies (DeepThinkVLA-SFT/RL, three seeds) both differences are significant. With the word-order check in Q2 it becomes 11/11 against each floor (8 and 7 Holm). A blind human annotation gives the same signs on all 8 ECoT-lineage configs.

Q2 — If the disputed pairs flip, which way does the aggregate verdict go? It strengthens.

  • The disputed pairs. Beyond the pairs the panel rejects, they are the 18% of scramble edits that reorder the directions of a compound MOVE ("move back and left" → "left and back move"). Judges read these as ordinary English and accept them, but in the LIBERO annotations this order carries information: both orders of each pair are common ("back and left" 9,947×, "left and back" 8,400×), and the first-named direction is the larger one on 130/130 steps. CoT-trained LoRA policies respond to it: reorders move their action on 0.90–1.00 of pairs vs 0.23–0.36 for other scramble pairs (no-CoT: 0.25 vs 0.14).
  • Direction of the effect. These pairs inflate the scramble floor; removing them sharpens the opposite verdict. Every step (three-seed DeepThinkVLA rows in parentheses):
vs paraphrase floorvs scramble floor
all pairs11/11 negative, 7 Holm (9)10/11 positive, 1 Holm (2)
panel-accepted pairs11/11 negative, 7 Holm (9)8/11 positive, 0 Holm (2), none significantly negative
panel-accepted, reorders excluded11/11 negative, 8 Holm (9)11/11 positive, 7 Holm (8)
  • Competent policies. DeepThinkVLA-SFT/RL stay significant against both floors in every row.

  • Matched contrast. Unchanged on the rule- and judge-validated subsets: direction_flip 11/11 (11 Holm), negation 10/11 (6).

Q3 — What makes direction_flip and negation special? They are the two families that edit only MOVE/SUBTASK, the same tags paraphrase_null edits, and they change the movement itself, so they are the most direct test of content. They also admit an exact meaning-preserving counterpart by construction:

  • direction_flip ↔ a same-size synonym swap in the same tags (exact on 890/890 samples);
  • negation ("do not X") ↔ "you should X" (exact on 1000/1000).

Q4 — Obstacles for the other five families, and results. We built approximately matched nulls for all five and ran the same paired contrast. The same five-judge panel, on their exact scored pairs, rates them preserved on 0.97–0.99 (ECoT render; 0.72–1.00 on DeepThinkVLA's compact one), and restricting the contrast to the pairs it accepts changes no family's Holm count.

  • Obstacle. For the object families (subject_swap, adversarial_plausible), a same-length, referent-preserving synonym is often unavailable in LIBERO scenes; per-tag exact matching reaches 27–48%. Verb, gripper and location nulls match almost exactly (92.5–100%).
  • Results. Content sensitivity is concentrated in edits that change the movement itself: direction_flip 11/11 (11 Holm), verb_swap 10/11 (9), negation 10/11 (6). Object, location and gripper edits seldom significantly beat their null: adversarial 7/11 (2), subject 4/11, gripper 2/11, location 2/11 (0 Holm each).
  • Why this supports the paper. Unmatched, every family moves the action often and so looks CoT-sensitive; once locus and size are matched, only movement edits remain above their null, the paper's thesis at finer grain. All seven families are reported, each with its own matched null, so no claim rests on two families alone.
Reviewer_gmXbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper questions whether action-change magnitude reliably measures chain-of-thought (CoT) faithfulness in vision-language-action models. It introduces CoT-Faith, a framework with 13 reasoning interventions, matched controls, and open- and closed-loop evaluations. Experiments on ECoT-bridge, its LoRA variants, and DeepThinkVLA show that faithfulness judgments depend on control selection, and that large action changes need not follow the edited reasoning. For the high-performing DeepThinkVLA variants, removing CoT or editing it within the task has little effect on LIBERO success, while swapping in another task's CoT does lower it. The authors recommend assessing directional alignment and task outcomes alongside action magnitude.

Critical strengths
  1. The paper addresses whether CoT reasoning genuinely influences VLA behavior, challenging the common assumption that action sensitivity implies reasoning faithfulness. This has important implications for VLA interpretability and evaluation.
  2. Under direction_flip, ECoT-bridge shows a 96.3% action-change rate but only an 11.7% direction-reversal rate, clearly showing that action changes do not necessarily reflect semantic alignment.
  3. Experiments on DeepThinkVLA show that modifying CoT can alter individual actions without substantially affecting LIBERO task success, providing behavioral evidence beyond open-loop metrics.
Critical weaknesses
  1. The paper assumes that faithful reasoning should lead to actions consistent with an edited CoT. Appendix B.6.2 rules out conflicts with the geometry fields inside the CoT, but not conflicts with the image or the task instruction. For example, changing "move left" to "move right" may contradict what the scene requires, and a well-grounded model might reasonably ignore such edits. Failure to follow an edited CoT therefore does not necessarily indicate unfaithfulness.
  2. The direction metric uses the cosine between full 3D translation vectors with a −0.5 threshold. If direction_flip reverses only one axis while the others stay the same, even a correct reversal can fail this criterion (e.g., (1,1,1) vs. (−1,1,1) gives cosine +1/3). This could underestimate alignment and calls into question ECoT-bridge's drop from first to seventh. An axis-conditioned evaluation would help.
  3. The paper shows that different metrics disagree, but does not establish which one reliably separates CoT-dependent from CoT-independent policies. The action-only LoRA control is fine-tuned from a CoT-trained model and may keep its sensitivity, and the rule-based oracle does not reflect neural policies. Comparing CoT-coupled and CoT-independent neural policies trained under matched conditions would be a stronger validation. As is, the work is more convincing as a critique of existing metrics than as a validated benchmark.
  4. The null controls rely on an LLM judge with no human validation, and syntactic scrambling may not preserve meaning. The matched-null contrast was adopted after the sweep, which raises post-hoc concerns, and the identity control is non-zero on Bridge V2 (0.074–0.227), as the authors acknowledge. Human validation and clearer statistical reporting are needed.
Questions for authors
  1. How do the authors distinguish genuine CoT faithfulness from blindly following an edited CoT that contradicts the visual observation or task instruction?
  2. Why use the cosine of the full 3D translation vector rather than the specific axis affected by direction_flip? Would an axis-conditioned score change the ECoT-bridge ranking?
  3. Could the authors validate the proposed metrics with neural policies explicitly trained to either rely on or ignore CoT?
4 comments

Thank you for the careful and constructive review. Your questions point to the tests that matter most; we ran each: a factorial that separates the image, instruction and CoT channels (Q1), the axis-conditioned direction score (Q2), a known-ground-truth test with policies trained to use or ignore their CoT (Q3), and validation of the controls on the exact scored pairs (W4). All support the paper's conclusions.

Q1/W1

If a policy ignored an edited CoT because it correctly trusts the scene, it should follow the CoT once the CoT agrees with the scene. We ran a 2×2×2 factorial on the same frames: {mirror the camera image} × {flip left/right in the instruction} × {flip left/right in every CoT field}. In the "all consistent" cell all three inputs point the same new way, so nothing conflicts. Lateral-action reversal rate:

policyimage onlyCoT onlyall consistent
DeepThinkVLA SFT / RL (LIBERO, competent)0.16 / 0.190.10 / 0.120.19 / 0.21
LoRA r=32 (LIBERO)0.020.550.57
ECoT-bridge (Bridge V2, in distribution)0.220.530.89

This distinguishes faithful use from blind following across three behaviours:

  • Competent DeepThinkVLA does not take its direction from the CoT, even when the CoT agrees with the scene: a consistent edit reverses it on 0.19–0.21, as often as mirroring the image alone (0.16–0.19); with its own self-generated CoT, a CoT-only flip reverses on 0%. Only the image moves its direction, and an empty CoT, which conflicts with nothing, leaves its closed-loop success unchanged.
  • LoRA follows blindly, even against the scene (0.55 CoT-only vs 0.02 image-only).
  • ECoT-bridge combines both channels (0.53 CoT-only, 0.89 consistent).

Our conclusions do not equate a declined edit with unfaithfulness; they rest on magnitude scores rating all three alike, as highly CoT-sensitive. We add the factorial to the evaluation protocol.

Q2/W2

We use the 3D cosine because MOVE lines often describe a compound direction ("move back and left"), so reversing the whole motion is the natural criterion. We also computed the axis-conditioned score you suggest: ECoT-bridge's F_dir rises from 0.117 to 0.338, but its null ceiling rises in step (0.047 → 0.184), and the LoRA variants remain at 0.56–0.77. The ranking is unchanged: ECoT-bridge stays 7th of 8, its correlation with the magnitude ranking is ρ = 0.429 (3D: 0.476), and 7/11 configurations clear their null under both.

Q3/W3

We ran this known-ground-truth test. From the released DeepThinkVLA-SFT, we continued training for 6,000 steps in two arms differing only in their CoT: coupled (each frame keeps its own CoT) and shuffled (each frame's CoT is replaced by a random other frame's, so it carries no information about the action).

  • Ground truth holds. Both arms stay competent (success 0.980 / 0.970, 200 episodes each). Random-token CoT costs the coupled arm 17.5 points and the shuffled arm 2: one reads its CoT channel, the other ignores it.
  • Magnitude and total-variation scores separate the arms (semantic-edit F 0.197 vs 0.011), giving ≈0 to the policy that ignores its CoT.
  • Direction-aware score does not separate them: it is ≈0 in both, as on every DeepThinkVLA checkpoint, since the coupled arm's action changes but does not reverse. Reading a CoT and following it are different properties; the protocol reports both.

On known ground truth, both levels of the benchmark behave as intended: open-loop magnitude and TV scores detect whether a policy reads its CoT, and closed-loop success shows whether outcomes depend on it. The shuffled arm, from the same checkpoint and budget, is the matched CoT-independent control you asked for.

W4

  • Controls. On the exact scored pairs, five gate-passing judges from four families judged each pair in both orders (Qwen2.5-7B/32B, gemma-2-27B, Mistral-Small-24B, Command-R+). Their majority rates paraphrase_null 1.00 and syntactic_scramble 0.94 meaning-preserving, and meaning-changing edits 0.00–0.04. We also exclude the 18% of scramble edits that reorder a compound MOVE, whose first-named direction dominates (130/130 LIBERO steps). The finding strengthens: 11/11 below the paraphrase floor (8 Holm) and 11/11 above the scramble floor (7 Holm). Keeping the reorders: 11/11 below and 8/11 above, none significantly below, and both competent policies stay significant against both floors. A blind human annotation gives the same signs on all 8 ECoT-lineage configs.
  • Primary statistic. Every matched null was fixed before any action was decoded with it, and the contrast is reported for all seven families, without selection. The headline rests on the two floors, whose opposite verdicts hold on the validated pairs above.
  • Identity control. The non-zero rate on Bridge V2 is capitalization ("The" → "the"); identical renders move the action on 0/81 samples.
  • Reporting. The revision adds a per-claim table of test, n, CI and Holm family.

Thanks for the detailed response and the new experiments. A few remaining points:

  1. Appendix B.4.1 now says ECoT-bridge’s low 3D score on LIBERO doesn’t mean it fails to read the CoT, yet the abstract and Fig. 1c still present the drop from first to seventh as a main finding.
  2. In Table 48 the matched contrast doesn’t separate the two arms on F\mathcal{F} (the CI includes zero). Fdir\mathcal{F}_{\text{dir}} is zero on all arms because the FAST head never reverses, so it is uninformative here rather than failing, as the conclusion describes it.
  3. For the controls, the rule check passes only 24% of scramble pairs, and adding Llama-3.3-70B brings the rate to 0.67. The rebuttal reports only the 0.94.

I will keep my score for now and look forward to the authors’ response.

Thank you for the follow-up. We address the three points in order; each revision below is in the updated PDF.

1. ECoT-bridge's rank. The first-to-seventh result is a statement about the instruments, and the in-distribution analysis you cite completes it rather than qualifying it. On LIBERO, a magnitude leaderboard ranks ECoT-bridge first because its action moves under 0.963 of direction flips; a direction-aware score ranks it seventh because the action rarely reverses (0.117, mean cosine +0.418); in distribution, the factorial shows that it combines CoT and image (lateral reversal 0.53 under a CoT-only flip, 0.89 under a scene-consistent one). Three instruments give three pictures of the same model, which is the paper's claim: no single edit score licenses a faithfulness ranking. The revised abstract and Fig. 1c caption now state that the reranking shows how much the leaderboard depends on the instrument, not that ECoT-bridge ignores its reasoning.

2. Table 48.

  • Matched contrast on F. On the coupled arm it is 0.026 [−0.066, 0.118]: on a binary exceedance over 300 frames it is underpowered. The same construction is clearly positive on the released checkpoint, which reads its CoT by the closed-loop test (random-token CoT costs it 18.5 points): 0.143 [0.083, 0.200]. On TV it separates the coupled arm from the shuffled one: 0.069 [0.042, 0.097] against 0.004.
  • F_dir. The revised conclusion now calls it uninformative on this head, since the FAST head never reverses the translation under any edit.

3. Which validation to trust. The paper's panel is five judges (Qwen2.5-7B, Qwen2.5-32B, gemma-2-27B, Mistral-Small-24B, Command-R+); the six-judge vote adds Llama-3.3-70B. The two lower rates come from the two validations that agree least with the blind human annotation of action, direction, object and gripper (MOVE reorders set aside, see below):

validationagrees with the annotatormeaning-preserving pairs rejectedmeaning-changing pairs accepted
five-judge majority (the paper's panel)0.973%5%
six-judge majority (panel + Llama-3.3-70B)0.8912%5%
rule check0.7725%11%
  • The rule check compares word positions, not meaning. It requires every object phrase and every verb–direction pair to keep its form and position. A word-order scramble breaks these rules by design, so the check rejects a quarter of the pairs the annotator reads as unchanged, yet it accepts a moved preposition that changes the object's role ("move to the microwave" → "move the microwave to"). It cannot serve as the standard: its 24% says how rarely a scramble leaves word positions intact.
  • Llama-3.3-70B errs where the other judges do not. On the cleanest controls, the 295 rule-verified paraphrase_null pairs of the ECoT render (single-word synonyms such as move→shift, every direction word unchanged), it rejects 21, against 0–8 for each of the other five. In 16 of its 21 reasons it names the edit a synonym substitution (8; "B uses synonyms … but describes the same objects and directions") or reports a direction change the edit does not make (11; "opposite directions"); the other five never name a synonym and report 6 direction changes between them. Adding it to the panel is what lowers the agreement with the annotator from 0.97 to 0.89 and the scramble rate to 0.67.
  • Without the disputed pairs, the verdict stays significant. Whichever validation chooses the pairs, the judges with or without Llama-3.3-70B or even the rule check, the semantic mean is below the paraphrase floor on 11/11 configurations and above the scramble floor on 11/11, and none is significant the other way (Holm-significant configurations in parentheses):
scramble pairs kept bybelow paraphrase floorabove scramble floor
five-judge majority (panel), MOVE reorders excluded11/11 (8)11/11 (7)
six-judge majority (panel + Llama-3.3-70B), MOVE reorders excluded11/11 (7)11/11 (6)
rule check alone11/11 (7)11/11 (6)

The revised table also lists stricter and looser cuts. Where it shows few or no Holm-significant configurations against the scramble floor, the cut either keeps the MOVE reorders, which inflate the floor (all pairs; judge majority with reorders), or keeps few scramble pairs, which leaves little power (87 of 299 when all five judges must agree). In every row the semantic mean stays above the scramble floor on at least 8 of 11.

MOVE reorders are set aside because they change the instructed movement: in the LIBERO annotations the first-named direction is the larger one on 130/130 steps. The annotation gives the same signs on all 8 ECoT-lineage configurations.

We hope this resolves the remaining points.

Thank you for the clarifications. The reframing of the ECoT-bridge result, the corrected description of Fdir\mathcal{F}_{\text{dir}}, and the robustness table across validations address my remaining points. I have raised my score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.