acceptodds
Under review as a conference paper at ICLR 2027

Gains Stay Where the Loss Went: A Channel-by-Condition Map of Physical-Plausibility Training in Vision-Language Models

Abstract

A vision-language model shown a shadow that points toward the light calls the picture physically possible. Fine-tuning on plausibility repairs this, and prior work reports the repair on one readout, usually an accuracy. On ImPhysible, a rendered testbed of matched normal and violated images, we read one model's judgment on four channels, a 0 to 100 score, the first-token logit, the generated verdict, and a probe on the hidden state. We vary which channel and which condition the training loss passed through. Every (channel, condition) cell the loss passed through rose, in every variant we trained. The cells it did not pass through rose, stayed at chance, or moved the other way, and no variant made them rise reliably. A score-order reward lifts score pair-rank accuracy from 0.469 to 0.576 and pushes the logit of the same answers from 0.504 to 0.456. An alignment loss that holds out one domain raises that domain's representation cosine to between 0.556 and 0.729 while its free answer stays between 0.463 and 0.500. Decide which channel you will read, put the loss through it, and test untrained conditions separately. One model, rendered stimuli, and an earlier data version behind part of the channel-axis evidence bound the claim.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, physical plausibility, readout channels, linear probing, fine-tuning, synthetic benchmark, cross-domain transfer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YbRZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a vision-language model on rendered scene judgments and compares several ways of measuring its responses: numerical scores, preferences between answer tokens, generated verdicts, and measures from internal representations. Training that rewards numerical scores improves their pooled ordering accuracy while generated verdicts worsen. Transfer varies across domains and viewpoints. These results are useful, but the stronger claim that improvement follows the loss in every combination of readout and condition is inconsistent with disclosed results. The actual readout procedures also need reconciliation.

Critical strengths

The paper provides concrete reasons to check several ways of measuring responses after training. The pooled score improvement and verdict decline occur across three training seeds. A preliminary comparison of score and verdict rewards from the same starting model and an intervention in one seed that changes only the score digits provide useful controls. The digit change lowers verdict accuracy from 81.6% to 74.0%, supporting an effect of the preceding score text in that check. Domain results also show uneven and unfavorable transfer.

The mathematical arguments are valid for fixed pairs. They explain why lower average loss or better alignment of internal representations need not improve decisions on those pairs, without establishing a causal theorem about where training effects must appear. The supplied summaries also reproduce the checked table values. The authors disclose earlier and current data versions, along with reliance on one model, rendered scenes and few seeds for some experiments. The current test sets also contain few distinct scenes; variation across seeds does not measure uncertainty over new scenes.

Critical weaknesses

The conclusion about individual cells overstates the evidence. Conditions are defined to include individual domains, yet score-ranking accuracy in the trained liquid domain falls from 53.7% to a three-seed average of 52.1%. Liquid token preferences and untrained size scores improve in all three saved seeds despite lacking direct training objectives. Partial viewpoint transfer also calls for a graded account. These disclosed exceptions challenge both “every cell rose” and an individual-cell reading of “only where the loss went.” The authors' statement that other cells can rise supports a pooled interpretation, provided the definitions and headline are revised together. The contribution should also be distinguished from earlier post-training representation/output comparisons, including Vompa et al. and Fu et al.

The score and token-preference definitions do not consistently match the released procedures. The text defines a possibility score with its direction reversed, while code requests and ranks raw impossibility scores. Applying opposite directions to the same responses reverses pair ordering. Token contrasts are also computed under different prefixes and positions, including after a generated answer. The archive omits raw response rows and pair manifests, and documented runners have broken imports. Some helpers may be legacy files, so these discrepancies do not establish that a historical table used the wrong procedure. They do prevent a clear interpretation without a record of which procedure produced each table.

Questions for authors

For each table, can you provide a record of the data version, checkpoint, prompts, training objectives and scoring procedure, with raw answers and working training and evaluation scripts? The record should specify which score direction counts as better and which token probabilities are compared, including their positions and preceding text. Which common scenes and baselines for each angle support the viewpoint comparison? Consistent variant labels and definitions of “cell” and “reliably” would help.

What equal-size assumption makes the perspective scenes impossible, and where is it stated to the model? Increasing a distant object's actual 3D size is physically possible without that constraint. Rotation is already excluded from the physical-ten aggregate; preserve the distinction between physical judgments and relation/control tasks in broader labels.

6 comments
AuthorsOpenReview ↗

Response to Reviewer YbRZ

We thank the reviewer for checking code and archive against the text. The errors found are real. Table numbers below follow the revision, where submitted Tables 4, 5, and 7 become Tables 9, 10, and 8.

Corrections to the whole paper. Two descriptions on axes the paper rests on did not match the code. Section 3, Appendix B, and Table 6 call the alignment variant representation-only, but its objective is the answer-text cross-entropy plus one term, L = L_SFT + λ · mean_k (1 - cos(Δμ_k, d)) with λ = 1. Table 6 says the leave-one-out variants trained on nine of twelve domains, but they trained on eleven, water and rotation included.

Checking every description against configs, code snapshots, manifests, and run logs found four more mismatches that change how a result is read. (a) The leave-one-out variants and the verdict-SFT checkpoint trained on an earlier data version and were tested on the current one, unstated in the text. (b) The score-reward rows of Tables 2 and 3 used an adapter trained on the earlier version, which the text stated for Table 2 only. The revision reads both tables from the current-version adapter, so those rows change, and we withdraw the sentence that the kind of loss picks the channel on an untrained task. (c) On some tables the logit sits at a position the training objective does not read, so Section 3 and Table 6 state again which cells the loss passed through. (d) Appendix D gave probe values for the connector output that were read at language layer 16. At the connector the rise is absent. The second reply will list every change.

1. Scope of the claim. "The conclusion about individual cells overstates the evidence." We agree. The liquid counterexample stands in the submitted table, and the revised Table 2 has one on shadow. The revision states the claim on pooled cells, reports domain cells as counts, defines "cell" in Sections 1 and 3, keeps one name per variant, and drops "reliably". Section 4 and Tables 2 and 3 list the cells that rose off the loss path, among them the liquid logit and the size score. The new title is "What Moves and What Does Not: A Channel-by-Condition Map of Physical-Plausibility Training in Vision-Language Models".

2. Score direction. The text defined "a possibility score with its direction reversed." The text was wrong and the code was right. The prompt asks how likely the scene is physically impossible, and the number is used as written (Section 3, Appendix B). Under this rule, from one verdict-SFT checkpoint and one GRPO setup (Appendix G, Table 8), both rewards raised the verdict above the start in all three seeds, only the score reward raised the pair ordering by the written number, and under the verdict reward that number stayed where the start left it, with its ordering reversed. The start is one SFT seed on the earlier data version.

3. Token contrasts. Contrasts use "different prefixes and positions." Correct. The logit of Table 2 is read after the generated answer, not at the first token, and other tables use other positions. Table 7 and Appendix B give the full prompt, answer format, token position, and decoding behind each table.

4. Viewpoint. "Which common scenes and baselines for each angle"? Every angle uses the same 12 unseen scenes, 120 pairs per angle, and the base model is measured at each angle (Table 10, Figure 6). The rows are now over three seeds, and Section 5 describes the transfer as graded, partial at 45 degrees and at base at 90 degrees.

5. Scenes. "[V]ariation across seeds does not measure uncertainty over new scenes." Agreed. Appendices A and H give the test scenes per domain and 95% intervals that resample whole test scenes, and a comparison is read by the paired interval over the same scenes. Four domains have one test scene each and no per-domain scene interval.

6. Perspective. "[W]here is it stated to the model?" The image is violated only if the two objects are the same size, which the prompt does not state. The revision reports perspective with rotation as relation tasks and keeps both out of the physical aggregate (Section 2).

7. Prior work. Section 6 states how we differ from Vompa et al. and Fu et al. We read the score, logit, and verdict of the same answer after each loss.

8. Archive. "The archive omits raw response rows and pair manifests." Our reproducibility statement overstated the upload. We have replaced the archive on this submission with the pair manifests, raw model rows, working scripts, and a per-table record of data version, manifest, checkpoint, prompt, objective, and scoring, with file hashes. It follows the table numbers and values of the submitted draft, and its README lists where text and records differ. Rows behind revised tables will come with the revision. Images, adapters, and most hidden-state caches are replaced by hashes. All values in it are preliminary.

We will post a second reply with values once the revision is up.

Thank you for checking the text against the code and narrowing the claim. The score direction is now clear. The corrections to the training objectives, data versions and measurement positions change how several results should be interpreted. The additional training runs and checks across scenes are useful, but I would like to assess the corrected tables with their corresponding raw answers before changing my recommendation. I understand that the replacement archive currently follows the submitted tables, with records for the revised results still to come.

AuthorsOpenReview ↗

Second response to Reviewer YbRZ: changes and values

The revision is up. As promised, this reply lists every change with its values. p. and L give page and line of the revised PDF. Submitted Tables 4, 5, and 7 are now Tables 9, 10, and 8.

Every change.

  1. Title, abstract, Figure 1, Sections 1 and 8. The claim covers pooled test pairs and variants that trained stably, and Table 11 reports a verdict reward that collapsed in two of three seeds. "Reliably" is gone. The submitted sentence that no variant raised an untrained verdict cell became false once the viewpoint rows had three seeds, so it now says the verdict rose above chance in all three seeds only at a camera turned by 45 degrees (L024, L530).
  2. Pool (L099-104). Perspective and rotation are relation tasks and water is flagged in the rendering audit, so the physical row pools nine domains, 360 pairs (submitted ten, 400).
  3. Table 1. DINOv2 remeasured on the current twelve domains. Three domains are at 1.000, support, thermal, and tower.
  4. Score-reward rows now come from the current-version adapter (Tables 2, 3, Figure 4). Pooled score 0.464 to 0.685 ± 0.036 (submitted 0.469 to 0.576), logit 0.499 to 0.385 ± 0.028 (0.504 to 0.456), verdict 0.525 to 0.360 ± 0.026 (0.519 to 0.457). Without support, thermal, and tower the logit fall does not survive, 0.524 ± 0.032 against 0.513, paired interval [-0.066, 0.127] (L248-250, Table 14). Section 4 is retitled accordingly.
  5. The corrections of our first reply: objective (L193-195, Appendix B), eleven domains (Tables 6 and 9), data versions (L754-759, Table 6), logit positions (L157-159, Table 7), connector probe (L913).
  6. New: Tables 4, 5, 7, 11, 12, 13, 14, Appendix B prompts, rewards, and schedules, the dose run of Appendix F, and the comparison with Fu et al. and Vompa et al. (L490-492). Three seeds now fill Tables 3, 8, 9, 10 and Figures 5 and 6. Appendix E states the positive-logit fact only for the two head-adapter seeds the rows confirm.
  7. Smaller edits: one name per variant, the cell defined at L049-051 and L186-188, a Table 6 footnote on where the alignment term reads the representation, test scenes and the highlight re-render in Appendix A (L677-690), the base probe 0.846 labelled as the twelve-domain pool (Appendix D), Figure 4b Spearman 0.833 and 0.486 (submitted 0.841 and 0.630), and two sentences removed, "A person sees the fault at once" and the closing "come next".
  8. Archive. Rows behind Tables 2, 3, 8, 9, 10, 12, 13, and 14, with a script that recounts their 675 printed cells.

1. Scope. Table 2 counts domains above base, 8, 3, and 1 of 9. The shadow score falls from 0.500 to 0.475 ± 0.045. Off the loss path the liquid logit rises from 0.450 to 0.717 ± 0.112 and highlight from 0.412 to 0.488 (L258-263). On the untrained size task the score reaches 0.558 ± 0.051 from 0.447 and the verdict 0.853, 0.500, and 0.594 from 0.500 (L265-267, Table 3, submitted 0.645 ± 0.130 and 0.526 ± 0.040).

2. Score direction. The score is used as written (L153-154, L778). From one verdict-SFT checkpoint at 0.200, the score reward reaches 0.838 ± 0.013 and the verdict reward 0.207 ± 0.016, while the verdict goes from 0.707 to 0.770 ± 0.034 and 0.767 ± 0.024 (Table 8). Over the same scenes the score moves +0.638 [0.492, 0.813] against +0.007 [-0.020, 0.042] (Table 13).

3. Token contrasts. Table 7 (p.15). On the score tables the logit is read after the generated answer, a newline, and "VALIDITY:". On Table 9 it is the first answer position. Table 4 reports no logit.

4. Viewpoint. Table 10 (p.20), three seeds, base at each angle: -45 degrees 0.639 ± 0.021 against 0.517, trained 0.903 ± 0.002 against 0.483, +45 0.593 ± 0.015 against 0.471, +90 0.517 ± 0.026 against 0.492. Submitted, one seed, 0.629, 0.900, 0.579, 0.508 and base 0.47 to 0.52. Section 5 (L420-422), Figure 6 with scene intervals.

5. Scenes. Table 13 (p.22), Appendix A (L677). Against base the score reward moves score +0.221 [0.124, 0.334], logit -0.114 [-0.214, -0.012], verdict -0.165 [-0.260, -0.081]. The held-out cosine interval reaches zero on liquid, [-0.128, 0.573], and not on tower, [0.650, 0.830] (L323).

6. Perspective. L099-104.

7. Prior work. L487-492.

8. Archive. Item 8 above.

Reviewer_mJytOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether improvements from physical plausibility training can transfer across different reading modalities within the same vision-language model. The study introduces ImPhysible, a dataset generated via procedural rendering that includes both physically plausible and implausible scenes, and evaluates Qwen2.5-VL-7B across four channels: generated numerical scores, logit comparisons of continuous answer tokens, generated judgment outputs, and representation layer diagnostics. Training variants apply supervision on different channels while excluding specific domains, viewpoints, or relational directions.

Critical strengths

1 The paper addresses a useful evaluation blind spot. Reporting a single post-training accuracy can conceal substantial differences between numerical judgments, continuous logits, and generated answers. Evaluating these readouts together provides a more informative picture of model behavior.

2 Several observed discrepancies are substantial. The opposing aggregate changes after score-reward training, together with the viewpoint and relation-direction results, demonstrate that gains on one evaluation configuration should not automatically be interpreted as broadly improved physical reasoning.

3 The paper carefully distinguishes several measurements that are often conflated. The separation between continuous ranking, thresholded decisions, and generated verdicts is particularly helpful.

Critical weaknesses

1 The comparisons do not sufficiently isolate the effect of the supervised channel. Score training uses GRPO, verdict training uses cross-entropy, and representation training uses an alignment objective. The results therefore combine changes in the target channel with changes in optimization and potentially prompt format and training budget. The shared-start reward comparison in Appendix G is useful, but remains a single-seed experiment on the earlier data version. A controlled comparison is needed before attributing the observed pattern primarily to where the loss is applied.

2 Most experiments use one training seed, and the main channel comparison relies on an earlier dataset version. The current-version checks in Appendix G are preliminary and use different starting conditions. Moreover, multiple violation levels share the same scene, so uncertainty should account for scene-level dependence. Without confidence intervals and repeated runs, claims that untrained cells remain “at chance” or fail to improve “reliably” are insufficiently established.

3 Important implementation and reporting details remain unclear. Appendix B does not provide enough information to reproduce the exact prompts, reward calculation, decoding settings, token positions, representation extraction, or optimization setup.

Questions for authors

1 Do the principal findings survive removal of background cues? Can the authors repeat the key comparisons with background masking or controlled background randomization, and report results separately for domains with strong background-only predictability?

2 How exactly are the score, logit, and verdict channels elicited? Please provide complete prompts, token positions, and whether the verdict is conditioned on previously generated score digits. Is the quoted score of 95 a raw possibility score or a transformed violation score?

3 What are the exact training and held-out domains in Table 4? Please clarify why the leave-one-out variants train on nine rather than eleven domains, and report pair-level answer-axis projections alongside the cosine of the mean difference.

4 Please correct some grammatical errors, such as "a variant" instead of "an variant".

3 comments
AuthorsOpenReview ↗

Response to Reviewer mJyt

We thank the reviewer for the specific questions. Most point to real gaps. Table numbers below follow the revision, where submitted Tables 4 and 7 become Tables 9 and 8.

Corrections to the whole paper. Two descriptions on axes the paper rests on did not match the code. Section 3, Appendix B, and Table 6 call the alignment variant representation-only, but its objective is the answer-text cross-entropy plus one term, L = L_SFT + λ · mean_k (1 - cos(Δμ_k, d)) with λ = 1. Table 6 says the leave-one-out variants trained on nine of twelve domains, but they trained on eleven, water and rotation included.

Checking every description against configs, code snapshots, manifests, and run logs found four more mismatches that change how a result is read. (a) The leave-one-out variants and the verdict-SFT checkpoint trained on an earlier data version and were tested on the current one, unstated in the text. (b) The score-reward rows of Tables 2 and 3 used an adapter trained on the earlier version, which the text stated for Table 2 only. The revision reads both tables from the current-version adapter, so those rows change, and we withdraw the sentence that the kind of loss picks the channel on an untrained task. (c) On some tables the logit sits at a position the training objective does not read, so Section 3 and Table 6 state again which cells the loss passed through. (d) Appendix D gave probe values for the connector output that were read at language layer 16. At the connector the rise is absent. The second reply will list every change.

W1. Isolating the channel. The comparison in Appendix G "remains a single-seed experiment on the earlier data version." Appendix G now reports it over three GRPO seeds (Table 8). The two arms differ only in the reward. Both raise the verdict above the start by about the same amount, only the score reward raises the ordering by the written number, and under the verdict reward that number stays where the start left it. The alignment variant's control is the verdict-SFT run of Table 9, with the same budget and image order and no added term. Appendix B states where answer format and budget differ between variants. The start remains one SFT seed on the earlier data version. No variant trains the alignment term alone.

W2. Seeds and scenes. "[U]ncertainty should account for scene-level dependence." Section 7 names the cells that remain one seed. Appendices A and H give the test scenes per domain and 95% intervals that resample whole test scenes, and a comparison is read by the paired interval over the same scenes. "Reliably" is dropped. Where a held-out verdict is said to stay at chance on liquid and tower, it is exactly 0.500 in all three seeds (Table 9).

W3. Details. Appendix B now gives the full prompt for every table, the reward terms and weights, greedy decoding and token limits, the logit position of every table (Table 7), where the probe and the cosine read the hidden state, and the training schedule of each variant.

Q1. Background cues. "Do the principal findings survive removal of background cues?" We repeated the main comparison with every test background filled with grey (Appendix H). Appendix H also pools the physical domains without the three a frozen DINOv2 feature separates at 1.000 and reports what changes, including findings that do not survive.

Q2. Elicitation. The quoted 95 is the raw integer under a prompt asking how likely the scene is impossible. The verdict is written after the digits in the same answer, and the logit of Table 2 is read after both (Section 3, Table 7). Pass criteria for a verdict-first variant over three seeds (new Table 4) were shared on October 22, before the runs. Its start prompt and reading order were adopted after the results, and from that start the comparability check failed, so we report both readings. Section 4 reports the coupling of number and verdict with no causal claim.

Q3. Leave-one-out domains. Eleven, as corrected above. Nine is only the reported set, the ten-domain aggregate of the submitted draft minus the held-out domain, and the caption of Table 9 now says so. Table 9 also gives, per seed, the share of held-out pairs that project toward the answer axis, with d′. On liquid fewer pairs project toward it than under base, so Section 5 says the liquid evidence depends on seed and scenes.

Q4. Grammar. Fixed.

Archive. We have replaced the archive on this submission with the pair manifests, raw model rows, working scripts, and a per-table record of data version, manifest, checkpoint, prompt, objective, and scoring, with file hashes. It follows the table numbers and values of the submitted draft, and its README lists where text and records differ. Rows behind revised tables will come with the revision. Images, adapters, and most hidden-state caches are replaced by hashes. All values in it are preliminary.

We will post a second reply with values once the revision is up.

Thank you for your reply. I keep my score.

AuthorsOpenReview ↗

Second response to Reviewer mJyt: changes and values

The revision is up. As promised, this reply lists every change with its values. p. and L give page and line of the revised PDF. Submitted Tables 4, 5, and 7 are now Tables 9, 10, and 8.

Every change.

  1. Title, abstract, Figure 1, Sections 1 and 8. The claim covers pooled test pairs and variants that trained stably, and Table 11 reports a verdict reward that collapsed in two of three seeds. "Reliably" is gone. The submitted sentence that no variant raised an untrained verdict cell became false once the viewpoint rows had three seeds, so it now says the verdict rose above chance in all three seeds only at a camera turned by 45 degrees (L024, L530).
  2. Pool (L099-104). Perspective and rotation are relation tasks and water is flagged in the rendering audit, so the physical row pools nine domains, 360 pairs (submitted ten, 400).
  3. Table 1. DINOv2 remeasured on the current twelve domains. Three domains are at 1.000, support, thermal, and tower.
  4. Score-reward rows now come from the current-version adapter (Tables 2, 3, Figure 4). Pooled score 0.464 to 0.685 ± 0.036 (submitted 0.469 to 0.576), logit 0.499 to 0.385 ± 0.028 (0.504 to 0.456), verdict 0.525 to 0.360 ± 0.026 (0.519 to 0.457). Without support, thermal, and tower the logit fall does not survive, 0.524 ± 0.032 against 0.513, paired interval [-0.066, 0.127] (L248-250, Table 14). Section 4 is retitled accordingly.
  5. The corrections of our first reply: objective (L193-195, Appendix B), eleven domains (Tables 6 and 9), data versions (L754-759, Table 6), logit positions (L157-159, Table 7), connector probe (L913).
  6. New: Tables 4, 5, 7, 11, 12, 13, 14, Appendix B prompts, rewards, and schedules, the dose run of Appendix F, and the comparison with Fu et al. and Vompa et al. (L490-492). Three seeds now fill Tables 3, 8, 9, 10 and Figures 5 and 6. Appendix E states the positive-logit fact only for the two head-adapter seeds the rows confirm.
  7. Smaller edits: one name per variant, the cell defined at L049-051 and L186-188, a Table 6 footnote on where the alignment term reads the representation, test scenes and the highlight re-render in Appendix A (L677-690), the base probe 0.846 labelled as the twelve-domain pool (Appendix D), Figure 4b Spearman 0.833 and 0.486 (submitted 0.841 and 0.630), and two sentences removed, "A person sees the fault at once" and the closing "come next".
  8. Archive. Rows behind Tables 2, 3, 8, 9, 10, 12, 13, and 14, with a script that recounts their 675 printed cells.

W1. Table 8 (p.19) now has three-seed rows from one verdict-SFT checkpoint at 0.200 / 0.803 / 0.707 (score / logit / verdict). The score reward gives 0.838 ± 0.013 / 0.827 ± 0.007 / 0.770 ± 0.034, the verdict reward 0.207 ± 0.016 / 0.833 ± 0.015 / 0.767 ± 0.024. Over the same 24 scenes the verdict rises by +0.063 [0.017, 0.118] and +0.060 [0.017, 0.114], and the score by +0.638 [0.492, 0.813] against +0.007 [-0.020, 0.042] (Table 13). The alignment control is the verdict-SFT column of Table 9 (L742).

W2. Section 7 (L508) lists the three-seed results. Table 13 (p.22) gives 24 pooled test scenes. Against base, the score reward moves score +0.221 [0.124, 0.334], logit -0.114 [-0.214, -0.012], and verdict -0.165 [-0.260, -0.081]. The held-out verdict is 0.500 on liquid and tower in all three seeds (Table 9).

W3. Appendix B (p.13-15): R = 1.0 r_order + 0.5 r_magnitude + 0.5 r_zero + 0.25 r_format, 500 steps, lr 1e-5, KL 0.04, eight samples. SFT runs 480 steps of 16 images at lr 2e-4. Greedy decoding with 32, 16, or 64 new tokens (L782), prompts verbatim (L802 on), probe at the last prompt token after the final norm (L784).

Q1. With every test background grey (Table 12, p.21) the score reward gives 0.634 / 0.485 / 0.431 against base 0.492 / 0.482 / 0.507. The score gain stays and the logit fall does not, as in item 4.

Q2. The score is used as written (L153-154). Within one answer, verdict impossible carries higher digits than verdict possible with probability at most 0.029 per seed (L252-256). In Table 4 the digit swap lowers the verdict from 0.770 ± 0.034 to 0.728 ± 0.019, and the verdict-first variant rises from 0.729 to 0.774 ± 0.017. Its score gain is 0.270 from its own prompt, under half of 0.638, and 0.619 from the score-first prompt, so both readings stay.

Q3. Table 9 caption (p.20). Share of held-out pairs projecting toward the axis (L1059-1061): liquid 0.588, 0.613, 0.675 against base 0.750, tower 0.775, 1.000, 1.000 against 0.525.

Q4. Fixed throughout.

Reviewer_9n6JOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies VLMs on the aspect of model improvement and degradation from a different loss formulation perspective. The authors choose physical plausibility judgment as a to evaluate the same VLM along channel and evaluation condition, where considered channels are a generated numerical plausibility score, the first-token logit difference between "impossible" and "possible", the generated binary verdict, and an internal representation readout using either a probe or cosine alignment. The main empirical finding is that cells corresponding to both a channel and condition through which the training objective passed improve, whereas cells outside that training path do not improve reliably, or degrade.

Critical strengths
  • The papers tries to understand an interesting question: what post-training improvement actually mean?
  • The paper is easy to read and clear on the central message authors want to convey.
Critical weaknesses
  • The paper achieves empirical understanding on the post-training behavior and the results are not surprising as well. The central weakness of this paper is lack of thorough experimentation. All the experiments are based on primarily using Qwen 2.5-VL-7B. The authors mention that second tested model exposes only a subset of channel. Without substantial experimental evidence, central claim remains weak.
  • All the primary variants freeze the vision encoder and adapt the language model. Its not clear from the experiments if the lack of condition generalization is an outcome from the training design choice.
  • It would have simple experiment to add to optimize multiple channels simultaneously. What happens if the objective is defined in such a way?
Questions for authors

Please address the weakness above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9n6J

We thank the reviewer for the review. The corrections below apply to the whole paper, so they come first.

Corrections to the whole paper. Two descriptions on axes the paper rests on did not match the code. Section 3, Appendix B, and Table 6 call the alignment variant representation-only, but its objective is the answer-text cross-entropy plus one term, L = L_SFT + λ · mean_k (1 - cos(Δμ_k, d)) with λ = 1. Table 6 says the leave-one-out variants trained on nine of twelve domains, but they trained on eleven, water and rotation included.

Checking every description against configs, code snapshots, manifests, and run logs found four more mismatches that change how a result is read. (a) The leave-one-out variants and the verdict-SFT checkpoint trained on an earlier data version and were tested on the current one, unstated in the text. (b) The score-reward rows of Tables 2 and 3 used an adapter trained on the earlier version, which the text stated for Table 2 only. The revision reads both tables from the current-version adapter, so those rows change, and we withdraw the sentence that the kind of loss picks the channel on an untrained task. (c) On some tables the logit sits at a position the training objective does not read, so Section 3 and Table 6 state again which cells the loss passed through. (d) Appendix D gave probe values for the connector output that were read at language layer 16. At the connector the rise is absent. The second reply will list every change.

W1. One model and the depth of evidence. "All the experiments are based on primarily using Qwen 2.5-VL-7B." We agree that one model bounds the claim. Section 7 keeps that limit, and the revision makes no claim beyond the model we tested. A second model we tried exposes only a head and a logit channel, so it cannot fill the map. We do not claim the result is surprising. What the map measures is that the score, logit, and verdict of the same answer need not move together after training, so one reported accuracy does not tell which readout rose. Within the tested model the revision adds three seeds to the shared-start comparison and the viewpoint table, scene-level intervals, a background-fill control, and a pool without the domains a frozen image feature already separates.

W2. Frozen encoder. The review asks "if the lack of condition generalization is an outcome from the training design choice." One correction to our own text. The vision blocks are frozen, but the connector between vision and language is trained in the main variants (Section 3, Appendix B). With the same frozen encoder, adding 20, 40, or 80 training pairs of a held-out domain raises the verdict on that domain (Appendix F, one seed), so the frozen encoder alone does not keep that verdict at chance. The revision does not claim that unfreezing would add nothing.

W3. Several channels at once. "What happens if the objective is defined in such a way?" Appendix G reports a reward through the score and the verdict together, and both cells it passed through rose, on one training seed. The shared-start comparison there, now over three seeds, separates the two single-channel rewards. From one verdict-SFT checkpoint and one GRPO setup, with every setting except the reward held fixed, the verdict rises above the start under both rewards, and the written number moves under the score reward only (Table 8). The joint reward has not been run over three seeds.

Title. The revision carries a new title that states the narrower claim, "What Moves and What Does Not: A Channel-by-Condition Map of Physical-Plausibility Training in Vision-Language Models".

Archive. We have replaced the archive on this submission with the pair manifests, raw model rows, working scripts, and a per-table record of data version, manifest, checkpoint, prompt, objective, and scoring, with file hashes. It follows the table numbers and values of the submitted draft, and its README lists where text and records differ. Rows behind revised tables will come with the revision. Images, adapters, and most hidden-state caches are replaced by hashes. All values in it are preliminary.

We will post a second reply with values once the revision is up.

AuthorsOpenReview ↗

Second response to Reviewer 9n6J: changes and values

The revision is up. As promised, this reply lists every change with its values. p. and L give page and line of the revised PDF. Submitted Tables 4, 5, and 7 are now Tables 9, 10, and 8.

Every change.

  1. Title, abstract, Figure 1, Sections 1 and 8. The claim covers pooled test pairs and variants that trained stably, and Table 11 reports a verdict reward that collapsed in two of three seeds. "Reliably" is gone. The submitted sentence that no variant raised an untrained verdict cell became false once the viewpoint rows had three seeds, so it now says the verdict rose above chance in all three seeds only at a camera turned by 45 degrees (L024, L530).
  2. Pool (L099-104). Perspective and rotation are relation tasks and water is flagged in the rendering audit, so the physical row pools nine domains, 360 pairs (submitted ten, 400).
  3. Table 1. DINOv2 remeasured on the current twelve domains. Three domains are at 1.000, support, thermal, and tower.
  4. Score-reward rows now come from the current-version adapter (Tables 2, 3, Figure 4). Pooled score 0.464 to 0.685 ± 0.036 (submitted 0.469 to 0.576), logit 0.499 to 0.385 ± 0.028 (0.504 to 0.456), verdict 0.525 to 0.360 ± 0.026 (0.519 to 0.457). Without support, thermal, and tower the logit fall does not survive, 0.524 ± 0.032 against 0.513, paired interval [-0.066, 0.127] (L248-250, Table 14). Section 4 is retitled accordingly.
  5. The corrections of our first reply: objective (L193-195, Appendix B), eleven domains (Tables 6 and 9), data versions (L754-759, Table 6), logit positions (L157-159, Table 7), connector probe (L913).
  6. New: Tables 4, 5, 7, 11, 12, 13, 14, Appendix B prompts, rewards, and schedules, the dose run of Appendix F, and the comparison with Fu et al. and Vompa et al. (L490-492). Three seeds now fill Tables 3, 8, 9, 10 and Figures 5 and 6. Appendix E states the positive-logit fact only for the two head-adapter seeds the rows confirm.
  7. Smaller edits: one name per variant, the cell defined at L049-051 and L186-188, a Table 6 footnote on where the alignment term reads the representation, test scenes and the highlight re-render in Appendix A (L677-690), the base probe 0.846 labelled as the twelve-domain pool (Appendix D), Figure 4b Spearman 0.833 and 0.486 (submitted 0.841 and 0.630), and two sentences removed, "A person sees the fault at once" and the closing "come next".
  8. Archive. Rows behind Tables 2, 3, 8, 9, 10, 12, 13, and 14, with a script that recounts their 675 printed cells.

W1. One model. Section 7 (L513) keeps the bound. Within the tested model the revision adds three seeds to the shared-start comparison and the viewpoint table (Tables 8 and 10), scene-level intervals over 24 pooled test scenes (Table 13), a background fill, and a pool without the three domains a frozen feature separates. With every test background grey the score reward gives 0.634 / 0.485 / 0.431 (score / logit / verdict) against base 0.492 / 0.482 / 0.507 (Table 12). Without those three domains the score rises from 0.479 to 0.584 ± 0.029, paired interval [0.035, 0.204], and the logit fall does not survive (Table 14). The viewpoint verdict at 45 degrees is 0.593 ± 0.015 and 0.639 ± 0.021 against base 0.471 and 0.517.

W2. Frozen encoder. The main variants are LoRAs on 198 modules, two of them the linear layers of the connector, with the vision blocks frozen (L196-198, L725). With the same frozen encoder, adding 20, 40, or 80 pairs of the held-out domain raises its free-answer pair-rank from 0.500 to 0.713, 0.812, and 0.775 on perspective, to 0.575, 0.650, and 0.650 on liquid, and to 1.000 on tower, while nine trained domains stay between 0.717 and 0.806, one seed (Appendix F, L973).

W3. Several channels. From the verdict-SFT checkpoint at 0.200 / 0.803 / 0.707, the joint reward through the score and the verdict reaches 0.844 / 0.822 / 0.785, one seed (last row of Table 8, p.19). Over three seeds the score reward gives 0.838 ± 0.013 / 0.827 ± 0.007 / 0.770 ± 0.034 and the verdict reward 0.207 ± 0.016 / 0.833 ± 0.015 / 0.767 ± 0.024. Over the same scenes the score moves +0.638 [0.492, 0.813] under the score reward and +0.007 [-0.020, 0.042] under the verdict reward (Table 13). The joint reward has not been run over three seeds.

Title. Item 1 above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.