- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CF-OPD to improve fine-grained visual recognition by incorporating controlled semantic edits into on-policy distillation. It constructs verified image pairs through controlled semantic edits and uses teacher distribution contrasts at student-generated prefixes to guide training on both original and edited images. Experiments on six visual benchmarks and a paired-edit evaluation set report improvements in recognition accuracy and responses to edited visual details compared with independent OPD baselines.
Critical strengths
- A targeted approach to fine-grained visual supervision. The paper addresses the challenge of recognizing answer-determining visual details in on-policy distillation by incorporating teacher distribution contrasts from paired, semantically edited images into the distillation targets. This provides a concrete training approach for improving sensitivity to relevant visual changes, rather than relying solely on independent supervision for individual images.
- Empirical improvements on standard and paired-image evaluations. Experiments across six visual benchmarks and two model scales report improvements over independent OPD baselines. The additional paired-image evaluation, using Flip and Blind metrics, directly assesses whether model predictions respond appropriately to changes in answer-relevant visual details. These results provide empirical evidence of the method's practical potential beyond improvements in conventional visual recognition accuracy.
Critical weaknesses
- Insufficient isolation of the core contrast mechanism. The central design of CF-OPD is to construct distillation targets using teacher log-probability contrasts between original and edited image crops. However, the reported experiments do not clearly isolate the contribution of this contrast signal by removing it while keeping the paired training data, rollout configuration, and compute budget fixed. Consequently, it remains difficult to determine whether the reported improvements arise specifically from the proposed contrast mechanism or from other factors, such as additional edited data and two-sided training.
- Insufficient justification of the counterfactual formulation and paired-data validity. The paper constructs image pairs through controlled semantic edits and applies pixel-level checks, answer verification, and logical audits to filter low-quality examples. While these procedures provide useful quality control, they do not fully rule out unintended semantic changes or editing artifacts that the student could exploit as shortcuts. As a result, the current evidence does not fully establish whether the student learns the intended relationship between answer-relevant visual attributes and predictions, rather than relying on incidental cues introduced by the editing process.
Questions for authors
- Can the authors provide a controlled ablation that removes the teacher log-probability contrast term while keeping the paired training data, rollout configuration, and compute budget fixed? If such an experiment has already been conducted, what are the resulting Avg₆, Flip, and Blind scores? This result would directly affect my assessment of whether the proposed contrast mechanism independently contributes to the reported gains.
- Can the authors provide additional evidence that the constructed image pairs change the intended answer-relevant attribute while preserving other relevant semantic content? In particular, is there any independent human evaluation or quantitative audit of unintended semantic changes or editing artifacts? Such evidence would materially affect my assessment of whether the observed gains can be attributed to the intended paired supervision.
7 comments
Response to Reviewer 3xuK — W1: Isolating the teacher contrast from data and two-sided training
↑: higher is better; ↓: lower is better.
Thank you for emphasizing the controlled test of the contrast mechanism. We added a strict contrast-removal experiment that retains the paired data, rollout procedure, and optimization budget, and repeated it with three seeds.
1. Removing only the contrast. Full CF-OPD and α=0 start from the same initialization and use the same mixed training data. Both independently generate responses on the original and edited members of each pair. The current and counterpart crops are scored by the same EMA teacher at the current student prefix. In the control, we still execute both teacher-view computations but multiply the log-probability contrast by zero. λ=0.15, candidate support, normalization, tail handling, and all other target settings are unchanged. This preserves the teacher computation used to construct the target in both configurations.
2. Matched training components. Additional edited examples and two-sided training are present in both configurations. Matching these components enables a direct test of the contrast contribution. The independent-OPD baseline provides a complementary same-input comparison; the strict α=0 comparison is the one used to isolate the contrast while retaining the mixture and two teacher views.
3. Full results across six recognition benchmarks and CF-Bench-X. Each method uses seeds 42, 43, and 44. Entries are mean±sample SD; scores are percentages.
| Metric | CF-OPD | α=0 | Independent OPD |
|---|---|---|---|
| V* ↑ | 93.72±0.91 | 90.23±0.60 | 90.58±0.91 |
| Zoom ↑ | 62.72±0.95 | 63.59±0.71 | 63.55±0.43 |
| HR-4K ↑ | 86.96±0.38 | 85.42±0.14 | 84.67±0.40 |
| HR-8K ↑ | 83.67±1.04 | 82.71±0.52 | 82.96±0.85 |
| MME-EN ↑ | 72.96±0.08 | 71.45±0.24 | 71.63±0.35 |
| MME-CN ↑ | 70.26±0.49 | 69.05±0.20 | 69.66±0.30 |
| Avg₆ ↑ | 78.3798±0.1206 | 77.0729±0.0963 | 77.1738±0.3444 |
| Flip ↑ | 75.85±0.39 | 71.76±1.18 | 72.38±0.51 |
| Blind ↓ | 17.42±0.68 | 19.78±1.24 | 19.44±0.79 |
Statistics and differences are calculated before rounding.
Against α=0, CF-OPD improves mean Avg₆ by 1.31 points, Flip by 4.09 points, and Blind by 2.35 points in the favorable direction. All three seeds improve these aggregate metrics. Mean accuracy is higher on five standard benchmarks, supporting the gain in the six-benchmark average. The independent-OPD comparison gives +1.21 Avg₆, +3.47 Flip, and −2.02 Blind points. These results demonstrate improvements in both conventional recognition and correct responses to edited details.
4. Mechanistic interpretation. The counterpart crop supplies an alternative visual observation at the same question and student prefix. The contrast changes token preferences according to how the teacher responds to that observation. The controlled results support a contribution from this target component beyond the benefit of merely exposing the model to more images or training on both sides. At inference, the model still uses only the full image; no counterfactual image or crop teacher is required.
Locations. §3.3–3.4, §4.4 Table 3, and Appendix A. The revised manuscript with changes highlighted in blue is in the Supplementary Material.
Response to Reviewer 3xuK — W2: Counterfactual validity, semantic preservation, and shortcuts
↑: higher is better; ↓: lower is better.
Thank you for highlighting the validity of counterfactual supervision. We added human evaluation of intended answer changes, editing quality, and visual artifacts, together with concrete examples and controlled training results.
1. What “counterfactual” specifies here. A pair is designed to change an answer-relevant visual attribute while keeping the question fixed between its two members. The intended answers differ. The corresponding evidence crops are scored by the same teacher at the same student-generated prefix; on the edited side, the crop roles are exchanged. This construction supplies an explicit alternative visual attribute and its corresponding answer for paired supervision.
2. Why the automatic checks need empirical validation. Pixel/locality checks constrain where changes occur, blind answer reading checks the resulting image–answer relation, and the logic audit checks execution and consistency. A localized edit can still change the wrong object, introduce a cue, or alter another relevant attribute. We therefore manually evaluated all 5,190 accepted training and test pairs, costing US$467.
| Criterion | Training (4,595) | Test (595) |
|---|---|---|
| Intended answer change ↑ | 4,227 (91.99%) | 559 (93.95%) |
| Editing problem ↓ | 207 (4.50%) | 12 (2.02%) |
| Visible artifact ↓ | 103 (2.24%) | 9 (1.51%) |
| Artifact may cue the answer ↓ | 81 (1.76%) | 6 (1.01%) |
The intended answer change is observed in 91.99% of training pairs and 93.95% of test pairs. The measurements separately identify unsuccessful/problematic edits, visible artifacts, and artifacts that may cue the answer. In the test set, visible artifacts occur in 1.51% of pairs and potentially answer-cueing artifacts in 1.01%.
3. Inspectable semantic-preservation failures. Appendix H shows the original image, editing instruction, edited image, and observed error for each example.
| Example | Requested edit | Observed failure |
|---|---|---|
| V* #145 | Move the shopping cart to the dog’s right | An extra cart appears on the left; the intended spatial answer does not change |
| HR-4K #522 | Move the bird onto the car roof | An inset introduces a different car, changing the object referent |
| HR-8K #458 | Turn the foreground child toward the camera | Another child is cut off and a bicycle disappears |
These cases, confirmed in the human evaluation, directly show that an edit can fail to change the intended answer or can modify non-target semantic content. Successful examples in Appendix F cover color, text, material, pattern, identity, and state changes. The examples make the observed editing outcomes inspectable, and the full-pool counts quantify their prevalence.
4. Evidence about what the student learns. Under fixed paired inputs, two-sided rollouts, and teacher-mixture settings, adding the contrast increases mean Flip from 71.76 to 75.85, reduces Blind from 19.78 to 17.42, and raises Avg₆ from 77.0729 to 78.3798. The improvement in six-benchmark average recognition accompanies the paired gain. Both configurations use the same edited images, so the comparison directly demonstrates the additional contribution of the contrast. The human quality measurements and controlled objective experiment jointly support the effectiveness of the paired supervision.
For the studied multimodal fine-tuning use, we consider the measured residual error proportions acceptable when pairs supplement original training examples. For paired evaluation, we also consider the measured error rates acceptable in this setting.
Locations. §3.2–3.4, §4.4 Table 3, Appendix G Table 6, and Appendices F/H. The blue-marked revision is in the Supplementary Material.
Response to Reviewer 3xuK
Thank you for the careful assessment and for recognizing the targeted visual supervision and complementary recognition and paired evaluations. We address each mechanism and data-validity concern below. The revised manuscript with changes highlighted in blue is provided in the Supplementary Material.