- Rating
- 2
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper shows hidden regressions arising from targeted fine-tuning: gains on the target task may be achieved at the expense of worse performance on non-target output fields. The authors show that such regressions can remain undetected by repair-matched validation or be obscured by the selected evaluation protocol. They additionally introduce DoseLint, a framework that selects targeted preservation tests by leveraging repair specifications and output dependencies.
Critical strengths
- This paper clearly identifies a failure mode in fine-tuning and presents an interesting obeservation regarding the discrepency between evaluation modes.
- This paper provides extensive controlled experiments with different strategies to try to address this failure.
Critical weaknesses
- The failure mode presented has limited novelty and generalizability. The tendency for fine-tuning on highly imbalanced data to bias model outputs toward dominant labels is largely foreseeable. Though the paper presents compelling failure examples, it lacks systematic cross-evaluation across model sizes, tasks and datasets to establish when and why these regressions emerge.
- The paper provides limited insights into mitigation. While it demonstrates severe regressions from imbalanced repair data, its best-performing training mitigation leverages data balancing, a well-established method. There is little analysis on why different mitigation approaches work or fail, and few new recommendations for preventing this failure mode in fine-tuning.
Questions for authors
Please see the concerns in the weakness.
Response to Reviewer Q3LG
Dear Reviewer Q3LG,
Thank you for highlighting the gap between evaluation modes and for asking what explains the results of the mitigation experiments.
Comment (1). “The tendency for fine-tuning on highly imbalanced data to bias model outputs toward dominant labels is largely foreseeable … it lacks systematic cross-evaluation across model sizes, tasks and datasets”
Response (1). We agree that a shift toward the dominant label is expected. Our contribution is to show how validation can miss the resulting loss. In the Qwen experiment, the forced-choice preservation check passed even as far-depth accuracy in free generation fell from 66.4% to 3.4–7.6% (§4.2).
We added two model families within CLEVR. In LLaVA-NeXT, we varied the share of rubber in the repair data across five settings. At the 0% and 100% endpoints, accuracy on the omitted material fell to zero in all three seeds. At 50%, correct metal answers ranged from 21 to 41 out of 42 in free generation. A prediction fitted at 25% and 75% gave 20 correct metal answers under forced choice at 50%, compared with an observed mean of 28.7 (Table G.4 and accompanying text).
The submitted Qwen2.5 sweep also contains two updates with the same 12.3-point target gain that left small-object size accuracy at 23% and 83% (Table C.5). Appendix C varies repair size, label share, learning rate, update type, and field pairs. The original CORD-v2 result provides a case where quantity accuracy largely held (Table F.1).
Comment (2). “little analysis on why different mitigation approaches work or fail”
Response (2). We agree that balancing is well known. We use it as a baseline mitigation to test whether preserving one field also preserves other fields. In Qwen, balancing raised far-depth generation accuracy to 93.0–95.4%, while the all-field check found that tie accuracy fell from 100% to 83.3% (Table 2 and §6.2).
The starting Qwen model already writes depth values outside the schema on 173 of 300 near scenes and 93 of 300 far scenes. The fixed and balanced repairs train the depth tokens directly, and these invalid values disappear in the mixed test. After masking removes that supervision, invalid values rise to 296 on the near half and 288–296 on the far half. Forced choice compares only allowed values, so it cannot reveal these schema errors (Appendix G.1).
To investigate masking, we fine-tuned two SmolVLM2 variants on the same rows with size excluded from the loss in both. Adding material supervision made 13 previously incorrect size answers on large objects correct in free generation across three seeds, with two changes in the opposite direction. Forced choice with the correct prefix still got 12 of those 13 cases wrong. Constraining generation to the two allowed sizes left those predictions unchanged. The predicted improvement on small objects was inconsistent (Appendix G.2).
All 500 responses per seed in the original Qwen masking runs parsed as JSON, but 92.4–96.6% of generated far-depth values were outside the allowed set (Table B.5). In LLaVA-NeXT, a logit offset chosen on 12 separate calibration scenes improved metal accuracy and lowered rubber accuracy in all three runs, with target accuracy unchanged (Appendix G.4). A bias refit in a smaller model likewise recovered only part of the lost accuracy after strong repairs and lowered accuracy on the value used in training (Appendix B.5).
We hope the new control clarifies how training on one field can change the generated answer in another, even when that field is masked. After mitigation, we recommend checking the generated answers for each relevant field and value against the same starting model.
Best,
The Authors