- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies representation gaps in vision-language models: a model often answers correctly from one equivalent representation of the same input but fails from another (e.g., a social-graph image vs. its edge list). Rather than proposing a new mechanism, it diagnoses a common training recipe — privileged-repair self-training, where a frozen teacher sees both a strong and a weak representation plus its two traces, rewrites the failed trace into a correct one, and a LoRA student trained only on the weak view matches the teacher's token distribution on that rewrite while replaying questions it already answers correctly. The paper makes three contributions. First, it introduces a method-agnostic diagnostic that decomposes training gains into reading of named regions, completion rather than truncation, and shifted answer priors. Second, it releases an open four-representation dataset across four domains (social graphs, chess, tables, molecules). Third, it reports a four-domain evaluation. Validation shows the recipe narrows gaps where the weak view is an image (chess image arm rises from 41.5 to 58.5 points) but helps little on tables and even widens the gap on molecules (SMILES barely moves). Gains transfer poorly, compare mixedly against answer-only SFT, and saturate after a second round. The contribution is diagnostic and cumulative, not mechanistic.
Critical strengths
The paper turns the vague question "does privileged self-training really close representation gaps" into a quantifiable protocol: a behavioral definition of the gap (K informationally equivalent representations, question-for-question), a three-way decomposition of training gains, and a released four-representation dataset across four domains. This is a contribution the community can adopt and cite independently of this paper's specific recipe.
Critical weaknesses
-
The statistical foundation is too thin for a diagnostic paper, which is the work's central claim. Every domain uses a single model and a single seed, with no variance, error bars, or significance testing (stated directly in §12); the anchor-loss evidence is admitted to be anecdotal (§6c), and several key deltas rest on single observations.
-
The method-agnostic claim and the baseline reference are not actually demonstrated. The protocol is presented as a reusable, method-independent diagnostic, yet it is exercised on only one instance — privileged-repair self-training — and is never run on the closest self-/context-distillation approaches that are discussed at length in related work. Meanwhile the only benchmarked alternative is answer-only SFT, explicitly the simplest, not the strongest, baseline.
Questions for authors
- Can you report multi-seed results (≥3 seeds) with mean ± variance for each domain, and run a significance test on the key deltas in Table 2 and Table 4 — most importantly the chess image arm (+17.00) and the SMILES arm (+0.71)? If the sub-1-point text-arm differences (e.g., edge list −0.11, prose +0.78) do not survive across seeds, which of the paper's conclusions would you retract?
- The anchor-loss evidence is described as anecdotal. Can you turn it into a quantifiable controlled comparison — for example, a matched run with anchor rows removed — so that the "anchors prevent forgetting" claim is supported by measurement rather than observation?
- To substantiate the method-agnostic claim, could you demonstrate the diagnostic protocol on a second instance, such as context distillation or on-policy self-distillation, rather than only on privileged-repair self-training?
- Can you benchmark at least one of the closest self-/context-distillation methods on the same platform (same model, same pair-mining, same scoring) alongside answer-only SFT, to position this variant within the method family rather than only against the simplest baseline?
4 comments
No anchor ablation
Q2. Controlled no-anchor ablation
TL;DR: We reran the Table 2 graph run with every anchor row removed and nothing else changed. Without anchors, accuracy improves (+1.33 global), but the model forgets: Across representations, the model consistently regresses performance on questions that model got right (shown in the table) Anchors therefore do what §6(b) claimed: they prevent forgetting without limiting the repair.
Setup. The two runs differ only in their training data: one includes the anchor rows, the other trains on the teacher-repaired rows alone. Model, LoRA configuration, loss, teacher contexts, optimiser, seed, batch size and epochs are unchanged, and each repaired row is seen exactly twice in both runs.
1. Accuracy improves without anchors
| Graph test accuracy (%) | No anchors | With anchors |
|---|---|---|
| Image | 72.33 | 71.22 |
| Adjacency list | 85.56 | 80.22 |
| Edge list | 86.44 | 86.22 |
| Prose | 86.33 | 87.67 |
| Global | 82.67 | 81.33 |
2. But the model forgets what it already knew
Of the items the base model answered correctly, the number each run now answers wrongly:
| Base-correct items | No anchors | With anchors |
|---|---|---|
| Image (n=570) | 43 | 39 |
| Adjacency list (n=733) | 51 | 17 |
| Edge list (n=777) | 56 | 16 |
| Prose (n=782) | 72 | 9 |
| All text (n=2,292) | 179 | 42 |
What we will change in the paper. We will replace the anecdotal §6(b) with this measurement:
(b) Anchors prevent off-target forgetting. In a matched ablation that removes the anchor rows and changes nothing else (Appendix X), headline accuracy rises by 1.33 points, but the model forgets as shown in the table, that shows the model regression on answers the base model already got right. Headline accuracy alone would therefore misreport what anchors do.
Appendix X will give the full tables and the setup.
Thanks for the additional experiments. I maintain my original score.
Thank you for reading the new results. If there are specific concerns that remain that you would like addressed further, we’d be grateful to know which ones so we can address them in the revision.We thank you for the constructive feedback that helped us improve our analysis.
More details responses to concerns raised
We thank the reviewer. Some experiments are in progress, but we are presenting some of the results we have analysed, so that the reviewer has a chance to read this. We will update this response with full response over the next few days, once we have all the results
In brief: the graph image gain, the global gain and the narrowing of the gap hold across three seeds;
W1/Q1. Multi-seed results and significance
Each seed changes the LoRA initialisation and the order of the training batches; Training targets are not regenerated, so the spread is training variance. All seeds are scored against the same frozen-base output graph test items.
Similarly, for chess, the multi seed run is as follows
What we retract or revise:
W2/Q3–Q4. Method-agnostic claim and stronger baselines
A STaR-style control (C2, graphs; full details in our response to Reviewer 1). It uses the same model, pair mining, data budget, KL objective and scoring, but the teacher's context holds only the gold answer, with no strong representation. We have listed a clear analysis shown here that shows that though StaR gets reasonable performance boost, slightly less than ours, its reasoning trace are factually wrong in a significantly proportionate way, which makes this method more desirable. More details can be found at response to the other reviewer https://openreview.net/forum?id=GJd4AVPEKT¬eId=j1WVAgYv9I (section - W1/Q1. Is the gain due to the privileged repair?)
Answer-only SFT (§8) is the simplest baseline and it does quite well, atleast as per accuracy (already tabulated in the paper) : in domain it leads ours on accuracy but a lot is attributed to truncation. The truncation diagnostic at https://openreview.net/forum?id=GJd4AVPEKT¬eId=j1WVAgYv9I (Section - W3/Q2. Truncation) already explains part of that lead, since SFT always answers. Ultimately, the core motivation of the method is to have reasoning traces from different representation guide the model towards stronger understanding of the representation in self repair way, so that we will get answer along with reasoning. So, we will concede in the paper that, if accuracy and the final answer, without reasoning, is all that really matter, the better choice is SFT, not our self repair framework