- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper trains a vision-language model on rendered scene judgments and compares several ways of measuring its responses: numerical scores, preferences between answer tokens, generated verdicts, and measures from internal representations. Training that rewards numerical scores improves their pooled ordering accuracy while generated verdicts worsen. Transfer varies across domains and viewpoints. These results are useful, but the stronger claim that improvement follows the loss in every combination of readout and condition is inconsistent with disclosed results. The actual readout procedures also need reconciliation.
Critical strengths
The paper provides concrete reasons to check several ways of measuring responses after training. The pooled score improvement and verdict decline occur across three training seeds. A preliminary comparison of score and verdict rewards from the same starting model and an intervention in one seed that changes only the score digits provide useful controls. The digit change lowers verdict accuracy from 81.6% to 74.0%, supporting an effect of the preceding score text in that check. Domain results also show uneven and unfavorable transfer.
The mathematical arguments are valid for fixed pairs. They explain why lower average loss or better alignment of internal representations need not improve decisions on those pairs, without establishing a causal theorem about where training effects must appear. The supplied summaries also reproduce the checked table values. The authors disclose earlier and current data versions, along with reliance on one model, rendered scenes and few seeds for some experiments. The current test sets also contain few distinct scenes; variation across seeds does not measure uncertainty over new scenes.
Critical weaknesses
The conclusion about individual cells overstates the evidence. Conditions are defined to include individual domains, yet score-ranking accuracy in the trained liquid domain falls from 53.7% to a three-seed average of 52.1%. Liquid token preferences and untrained size scores improve in all three saved seeds despite lacking direct training objectives. Partial viewpoint transfer also calls for a graded account. These disclosed exceptions challenge both “every cell rose” and an individual-cell reading of “only where the loss went.” The authors' statement that other cells can rise supports a pooled interpretation, provided the definitions and headline are revised together. The contribution should also be distinguished from earlier post-training representation/output comparisons, including Vompa et al. and Fu et al.
The score and token-preference definitions do not consistently match the released procedures. The text defines a possibility score with its direction reversed, while code requests and ranks raw impossibility scores. Applying opposite directions to the same responses reverses pair ordering. Token contrasts are also computed under different prefixes and positions, including after a generated answer. The archive omits raw response rows and pair manifests, and documented runners have broken imports. Some helpers may be legacy files, so these discrepancies do not establish that a historical table used the wrong procedure. They do prevent a clear interpretation without a record of which procedure produced each table.
Questions for authors
For each table, can you provide a record of the data version, checkpoint, prompts, training objectives and scoring procedure, with raw answers and working training and evaluation scripts? The record should specify which score direction counts as better and which token probabilities are compared, including their positions and preceding text. Which common scenes and baselines for each angle support the viewpoint comparison? Consistent variant labels and definitions of “cell” and “reliably” would help.
What equal-size assumption makes the perspective scenes impossible, and where is it stated to the model? Increasing a distant object's actual 3D size is physically possible without that constraint. Rotation is already excluded from the physical-ten aggregate; preserve the distinction between physical judgments and relation/control tasks in broader labels.
6 comments
Thank you for checking the text against the code and narrowing the claim. The score direction is now clear. The corrections to the training objectives, data versions and measurement positions change how several results should be interpreted. The additional training runs and checks across scenes are useful, but I would like to assess the corrected tables with their corresponding raw answers before changing my recommendation. I understand that the replacement archive currently follows the submitted tables, with records for the revised results still to come.
Second response to Reviewer YbRZ: changes and values
The revision is up. As promised, this reply lists every change with its values. p. and L give page and line of the revised PDF. Submitted Tables 4, 5, and 7 are now Tables 9, 10, and 8.
Every change.
- Title, abstract, Figure 1, Sections 1 and 8. The claim covers pooled test pairs and variants that trained stably, and Table 11 reports a verdict reward that collapsed in two of three seeds. "Reliably" is gone. The submitted sentence that no variant raised an untrained verdict cell became false once the viewpoint rows had three seeds, so it now says the verdict rose above chance in all three seeds only at a camera turned by 45 degrees (L024, L530).
- Pool (L099-104). Perspective and rotation are relation tasks and water is flagged in the rendering audit, so the physical row pools nine domains, 360 pairs (submitted ten, 400).
- Table 1. DINOv2 remeasured on the current twelve domains. Three domains are at 1.000, support, thermal, and tower.
- Score-reward rows now come from the current-version adapter (Tables 2, 3, Figure 4). Pooled score 0.464 to 0.685 ± 0.036 (submitted 0.469 to 0.576), logit 0.499 to 0.385 ± 0.028 (0.504 to 0.456), verdict 0.525 to 0.360 ± 0.026 (0.519 to 0.457). Without support, thermal, and tower the logit fall does not survive, 0.524 ± 0.032 against 0.513, paired interval [-0.066, 0.127] (L248-250, Table 14). Section 4 is retitled accordingly.
- The corrections of our first reply: objective (L193-195, Appendix B), eleven domains (Tables 6 and 9), data versions (L754-759, Table 6), logit positions (L157-159, Table 7), connector probe (L913).
- New: Tables 4, 5, 7, 11, 12, 13, 14, Appendix B prompts, rewards, and schedules, the dose run of Appendix F, and the comparison with Fu et al. and Vompa et al. (L490-492). Three seeds now fill Tables 3, 8, 9, 10 and Figures 5 and 6. Appendix E states the positive-logit fact only for the two head-adapter seeds the rows confirm.
- Smaller edits: one name per variant, the cell defined at L049-051 and L186-188, a Table 6 footnote on where the alignment term reads the representation, test scenes and the highlight re-render in Appendix A (L677-690), the base probe 0.846 labelled as the twelve-domain pool (Appendix D), Figure 4b Spearman 0.833 and 0.486 (submitted 0.841 and 0.630), and two sentences removed, "A person sees the fault at once" and the closing "come next".
- Archive. Rows behind Tables 2, 3, 8, 9, 10, 12, 13, and 14, with a script that recounts their 675 printed cells.
1. Scope. Table 2 counts domains above base, 8, 3, and 1 of 9. The shadow score falls from 0.500 to 0.475 ± 0.045. Off the loss path the liquid logit rises from 0.450 to 0.717 ± 0.112 and highlight from 0.412 to 0.488 (L258-263). On the untrained size task the score reaches 0.558 ± 0.051 from 0.447 and the verdict 0.853, 0.500, and 0.594 from 0.500 (L265-267, Table 3, submitted 0.645 ± 0.130 and 0.526 ± 0.040).
2. Score direction. The score is used as written (L153-154, L778). From one verdict-SFT checkpoint at 0.200, the score reward reaches 0.838 ± 0.013 and the verdict reward 0.207 ± 0.016, while the verdict goes from 0.707 to 0.770 ± 0.034 and 0.767 ± 0.024 (Table 8). Over the same scenes the score moves +0.638 [0.492, 0.813] against +0.007 [-0.020, 0.042] (Table 13).
3. Token contrasts. Table 7 (p.15). On the score tables the logit is read after the generated answer, a newline, and "VALIDITY:". On Table 9 it is the first answer position. Table 4 reports no logit.
4. Viewpoint. Table 10 (p.20), three seeds, base at each angle: -45 degrees 0.639 ± 0.021 against 0.517, trained 0.903 ± 0.002 against 0.483, +45 0.593 ± 0.015 against 0.471, +90 0.517 ± 0.026 against 0.492. Submitted, one seed, 0.629, 0.900, 0.579, 0.508 and base 0.47 to 0.52. Section 5 (L420-422), Figure 6 with scene intervals.
5. Scenes. Table 13 (p.22), Appendix A (L677). Against base the score reward moves score +0.221 [0.124, 0.334], logit -0.114 [-0.214, -0.012], verdict -0.165 [-0.260, -0.081]. The held-out cosine interval reaches zero on liquid, [-0.128, 0.573], and not on tower, [0.650, 0.830] (L323).
6. Perspective. L099-104.
7. Prior work. L487-492.
8. Archive. Item 8 above.
Response to Reviewer YbRZ
We thank the reviewer for checking code and archive against the text. The errors found are real. Table numbers below follow the revision, where submitted Tables 4, 5, and 7 become Tables 9, 10, and 8.
Corrections to the whole paper. Two descriptions on axes the paper rests on did not match the code. Section 3, Appendix B, and Table 6 call the alignment variant representation-only, but its objective is the answer-text cross-entropy plus one term, L = L_SFT + λ · mean_k (1 - cos(Δμ_k, d)) with λ = 1. Table 6 says the leave-one-out variants trained on nine of twelve domains, but they trained on eleven, water and rotation included.
Checking every description against configs, code snapshots, manifests, and run logs found four more mismatches that change how a result is read. (a) The leave-one-out variants and the verdict-SFT checkpoint trained on an earlier data version and were tested on the current one, unstated in the text. (b) The score-reward rows of Tables 2 and 3 used an adapter trained on the earlier version, which the text stated for Table 2 only. The revision reads both tables from the current-version adapter, so those rows change, and we withdraw the sentence that the kind of loss picks the channel on an untrained task. (c) On some tables the logit sits at a position the training objective does not read, so Section 3 and Table 6 state again which cells the loss passed through. (d) Appendix D gave probe values for the connector output that were read at language layer 16. At the connector the rise is absent. The second reply will list every change.
1. Scope of the claim. "The conclusion about individual cells overstates the evidence." We agree. The liquid counterexample stands in the submitted table, and the revised Table 2 has one on shadow. The revision states the claim on pooled cells, reports domain cells as counts, defines "cell" in Sections 1 and 3, keeps one name per variant, and drops "reliably". Section 4 and Tables 2 and 3 list the cells that rose off the loss path, among them the liquid logit and the size score. The new title is "What Moves and What Does Not: A Channel-by-Condition Map of Physical-Plausibility Training in Vision-Language Models".
2. Score direction. The text defined "a possibility score with its direction reversed." The text was wrong and the code was right. The prompt asks how likely the scene is physically impossible, and the number is used as written (Section 3, Appendix B). Under this rule, from one verdict-SFT checkpoint and one GRPO setup (Appendix G, Table 8), both rewards raised the verdict above the start in all three seeds, only the score reward raised the pair ordering by the written number, and under the verdict reward that number stayed where the start left it, with its ordering reversed. The start is one SFT seed on the earlier data version.
3. Token contrasts. Contrasts use "different prefixes and positions." Correct. The logit of Table 2 is read after the generated answer, not at the first token, and other tables use other positions. Table 7 and Appendix B give the full prompt, answer format, token position, and decoding behind each table.
4. Viewpoint. "Which common scenes and baselines for each angle"? Every angle uses the same 12 unseen scenes, 120 pairs per angle, and the base model is measured at each angle (Table 10, Figure 6). The rows are now over three seeds, and Section 5 describes the transfer as graded, partial at 45 degrees and at base at 90 degrees.
5. Scenes. "[V]ariation across seeds does not measure uncertainty over new scenes." Agreed. Appendices A and H give the test scenes per domain and 95% intervals that resample whole test scenes, and a comparison is read by the paired interval over the same scenes. Four domains have one test scene each and no per-domain scene interval.
6. Perspective. "[W]here is it stated to the model?" The image is violated only if the two objects are the same size, which the prompt does not state. The revision reports perspective with rotation as relation tasks and keeps both out of the physical aggregate (Section 2).
7. Prior work. Section 6 states how we differ from Vompa et al. and Fu et al. We read the score, logit, and verdict of the same answer after each loss.
8. Archive. "The archive omits raw response rows and pair manifests." Our reproducibility statement overstated the upload. We have replaced the archive on this submission with the pair manifests, raw model rows, working scripts, and a per-table record of data version, manifest, checkpoint, prompt, objective, and scoring, with file hashes. It follows the table numbers and values of the submitted draft, and its README lists where text and records differ. Rows behind revised tables will come with the revision. Images, adapters, and most hidden-state caches are replaced by hashes. All values in it are preliminary.
We will post a second reply with values once the revision is up.