- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
By finetuning vision language action models (VLAs) on LIBERO and CALVIN benchmark, the paper studies the linear decodability of web pretrained vision language models' (VLMs) pooled features. While a frozen VLM encodes action information loosely across high dimensional feature, finetuning increases the decodability and concentration in leading principal directions. During evaluation, this is supported by keeping the top 64 of 2048-4096 dimensions still retaining 88-96% success. However, controlled input intervention like swapping texture or single prompt token lead to task-selective failures while top-8 readability on LIBERO barely changes, showing that action decodability does not guarantee reliability in closed-loop control in full-finetuned policies (3e-5 on LIBERO, 1e-5 on CALVIN).
Critical strengths
- The paper studies the decodability and behavior reliability. The finding that the top-8 readability moves by only 0.02-0.05 while performance drops by about 48 and 60 points on LIBERO (readability was not measured under the CALVIN texture swap, where two skills drop 100%) is an interesting finding. This could be helpful for many literatures in this field that moves toward linear probing based fine-tuning.
- The experiments are well designed, and studied under controlled settings, supporting the concentration finding in the configurations studied.
- The experiments studied one visual intervention (texture swap) and one textual one (prompt change), showing that input-selective failures occur for both image and text interventions, in different benchmarks and policies.
Critical weaknesses
- The paper relies only on action chunk ridge as the decodability metric, other settings like per task or instruction are not studied systematically.
- The results rely on only a few runs and limited setups. The paper only studied one intervention for each of image and instruction inputs, and used only 1-2 seeds for the full-FT experiments in section 6, on two benchmarks only, with no confidence intervals, which weakens claims like "Readable is not enough".
- "Held-out episodes" were part of the backbone’s finetuning data. The authors admit this in the paper, but many probing results that directly supports the central claim rely on this setup, and this overlap could lead to backbone memorizing these data during training, and limits what the probe shows about generalization.
- While the paper shows that decodability does not equal reliability, both healthy and sensitive policies are concentrated and readable, neither an explanation of what separates them nor a better diagnostic is proposed.
- Many important experiments and results heavily rely on the appendix, making the scope of the paper broad and hard to follow, the presentation of the paper needs improvement.
- The title of the PDF differs from the one in the metadata
Questions for authors
- The held-out data were part of the backbone’s finetuning data, and the unseen-task probes show much lower (around 0.5-0.67). The main concentration claim ( 0.95) stated on in-distribution frames is not shown to hold outside the finetuning distribution. Can you show concentration (top-8 vs. full) on unseen tasks with multiple seeds and confidence intervals?
- Can the authors run on more seeds and setups? In section 6.2, the two seeds differ by about 20 points (94.8 vs. 74.9) under the training prompt and about 32 (46.9 vs. 15.0) under 'next 10'. No learning-rate search was run, and 1e-5 works while 3e-5 does not. This could mean the current hyperparameter setups are not well configured, or the results are not statistically rigorous to support the claims.
- Can you also analyze the readability measurement under the CALVIN texture swap? Readability was only measured under the LIBERO prompt change, so the claim "readable is not enough" is only partially tested for the visual intervention.
2 comments
Response to Reviewer 8uaR (2/2)
Q3: readability under the CALVIN texture swap. We ran this after submission (registered 12 October): the readout under the texture swap, on 1,558 recorded scene-D states, each rendered with the table's own texture and with table A's. The result: under the texture that breaks the policy, the action stays readable to a readout refit on the changed input, while a readout held fixed, and the policy's own head, lose it on the affected skills. In detail, each pair gives the fragile Emb-2B policy on the slider and light frames, first with the texture under which it solves 0 of the 226 such subtasks, then with the one under which it solves 223.
- A readout refit under each texture: 0.36 against 0.40 (0.47 under both over all frames).
- A readout fitted once on training-scene frames and held fixed: 0.18 against 0.40.
- The error of the policy's own head against the demonstrated actions: 0.19 against 0.09.
For the nine working checkpoints none of the three moves by more than 0.011. The visual intervention thus shows what the prompt intervention showed. One caveat, which a pre-registered rule required us to state: the fragile policies are less readable overall (34% and 10% from eight directions, against 46–52%), so readability ranks policies but does not flag the inputs that break them (Sec. 6, Appendix D.4). The readouts by skill and the fixed readout are reported in our replies; the revised PDF has the value over all frames and the head's error (Appendix D.4).
Your weaknesses, briefly.
- W1 (one metric). The revision reports the readout in three further ways: without the gripper (the eight directions still explain 80–89% on LIBERO, but only 1–14% of the rotations), with each task's average removed (93–95%), and per skill under the texture swap (Q3) (Sec. 4, Fig. 2, Appendices B.3 and B.4).
- W2 (few runs and settings, no intervals). Agreed for the submission; what we added and what is still thin is under Q2. The scope remains two simulated benchmarks and one robot arm (Sec. 7).
- W3 (held-out episodes belong to the backbone's training data). Agreed: they are new to the readout, not to the backbone. Settings in which the backbone never saw the data, unseen tasks and an unseen scene, are in Table 2; see Q1.
- W4 (what separates the policies; a better diagnostic). The recipe separates them (Q2). Two quantities register the failure where a refit readout does not: the head's own error on the affected skills (Q3), and a pre-registered measure of how far the input change moves the feature within the readout's directions, which separates three sensitive checkpoints from nine working ones without overlap. With so small a set we call it a candidate (Sec. 6, Appendix D.4).
- W5 (reliance on the appendix). The submission was hard to follow because of how it was written, and we apologize. The revision moves the evidence each claim rests on into the main text and rebuilds the appendix into six sections that follow it.
- W6 (title). The revised PDF returns to the title registered on OpenReview (general comment).
Response to Reviewer 8uaR (1/2)
Thank you very much for the careful review, and for your patience with a submission that was hard to follow; it matters a great deal to us. As our general comment explains, the revision is a full rewrite, and we do not ask for a re-evaluation on its basis. We answer your three questions first, each with what we ran and what it showed, and then the weaknesses briefly.
Q1: concentration outside the finetuning distribution. You ask for the eight leading directions against the whole feature on unseen tasks, with several seeds and intervals. We ran this after your review (registered 10 November): two further training seeds of the adapter that never saw the ten tasks, read in the same way, with intervals over test episodes. The result: outside the training data the readable share drops, by a third when the readout is fit on the new tasks and by more than half when it is not, but what remains still sits in the eight leading directions, for all three seeds. In detail (each pair is the R² from all of z / from its eight leading directions):
The revision therefore says that the concentration carries beyond the training data only in part, and only within the adapter's task family (Sec. 4, Table 2). A second adapter, trained jointly on CALVIN and the thirty tasks, has one seed (0.67 / 0.61 and 0.49 / 0.48 in the two settings). Table 2 gives the mean and range over the three seeds; Appendix B.5 gives each seed with its interval.
Q2: more seeds and settings; no learning-rate search. We ran this after submission (registered 10 October): a sweep of the batch and the learning rate of full finetuning on CALVIN, six policies, each with the data budget of the fragile policy. The result: your suspicion was right, the configuration was untuned. The fragile policies come from a learning rate too high for the batch; with a tuned rate, full finetuning matches LoRA and the texture no longer matters. In detail, subtasks completed in a row on the unseen scene D, over 1,000 chains, with 95% chain-bootstrap intervals:
Lowering only the learning rate of the original recipe to a tenth is enough (0.95 to 4.26). The texture swap changes the slider, light and LED subtasks of these six policies by at most 0.9 points, and three LoRA seeds solve all of them in every combination (Sec. 6, Table 4, Appendix D.5).
Three things remain thin. LIBERO has no sweep: full finetuning at 1e-5 works there (97.9%), and under the training prompt the two 3e-5 seeds differ mainly on the long-horizon suite (434 against 169 of 500), which we cannot explain. The 46.9% and 15.0% were measured under an evaluation prompt that differed from the training prompt; the revision reports the training-prompt values (94.8% and 74.9%) and keeps the other as a sensitivity test, which is positional (one extra space before the "8" does what "next 10" does). The fragile recipe was trained once per backbone on CALVIN, and the Instruct-2B policy was not re-tuned.