acceptodds
Under review as a conference paper at ICLR 2027

Hitchhiking Labels: When Repair Validation Misses Collateral Damage

22 views

Abstract

Targeted fine-tuning is often used to repair a narrow model failure, and is typically validated using data generated from the same specification as the repair data. We show that this practice can hide regressions in other fields of a structured output. When a repair specification fixes an input factor, another output field may receive the same label on every repair example—a hitchhiking label—while matched validation never tests the alternative factor values on which that field should change. In CLEVR-Fields, a Qwen3.5-9B repair improves such a field from 83.7% to 100% on repair-matched inputs while reducing accuracy on alternate inputs from 86.0% to 0%; the collapse repeats from the public base model across three seeds. We further find that whether a repair appears safe depends on how it is evaluated. Removing the affected field from the fine-tuning loss passes a forced-choice preservation check, yet its accuracy on alternate inputs is only 3.4–7.6% when the model generates the full structured output, with most errors coming from values outside the permitted schema. We introduce DoseLint, which uses logged repair factors, reference data, and output dependencies to request preservation tests on the missing input comparisons and in the model's intended output mode. In our approval study, given the affected field, 59-row tests on the omitted factor values detect 98.6% of its regressions, compared with 30.3% for tests sampled from the original data distribution. Repair validation should therefore account jointly for the inputs omitted by the repair specification and the way affected outputs are actually produced.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
corrective fine-tuning, backward-compatible model updates, regression testing, evaluation methodology, structured outputs, validation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Q3LGOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows hidden regressions arising from targeted fine-tuning: gains on the target task may be achieved at the expense of worse performance on non-target output fields. The authors show that such regressions can remain undetected by repair-matched validation or be obscured by the selected evaluation protocol. They additionally introduce DoseLint, a framework that selects targeted preservation tests by leveraging repair specifications and output dependencies.

Critical strengths
  1. This paper clearly identifies a failure mode in fine-tuning and presents an interesting obeservation regarding the discrepency between evaluation modes.
  2. This paper provides extensive controlled experiments with different strategies to try to address this failure.
Critical weaknesses
  • The failure mode presented has limited novelty and generalizability. The tendency for fine-tuning on highly imbalanced data to bias model outputs toward dominant labels is largely foreseeable. Though the paper presents compelling failure examples, it lacks systematic cross-evaluation across model sizes, tasks and datasets to establish when and why these regressions emerge.
  • The paper provides limited insights into mitigation. While it demonstrates severe regressions from imbalanced repair data, its best-performing training mitigation leverages data balancing, a well-established method. There is little analysis on why different mitigation approaches work or fail, and few new recommendations for preventing this failure mode in fine-tuning.
Questions for authors

Please see the concerns in the weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Q3LG

Dear Reviewer Q3LG,

Thank you for highlighting the gap between evaluation modes and for asking what explains the results of the mitigation experiments.

Comment (1). “The tendency for fine-tuning on highly imbalanced data to bias model outputs toward dominant labels is largely foreseeable … it lacks systematic cross-evaluation across model sizes, tasks and datasets”

Response (1). We agree that a shift toward the dominant label is expected. Our contribution is to show how validation can miss the resulting loss. In the Qwen experiment, the forced-choice preservation check passed even as far-depth accuracy in free generation fell from 66.4% to 3.4–7.6% (§4.2).

We added two model families within CLEVR. In LLaVA-NeXT, we varied the share of rubber in the repair data across five settings. At the 0% and 100% endpoints, accuracy on the omitted material fell to zero in all three seeds. At 50%, correct metal answers ranged from 21 to 41 out of 42 in free generation. A prediction fitted at 25% and 75% gave 20 correct metal answers under forced choice at 50%, compared with an observed mean of 28.7 (Table G.4 and accompanying text).

The submitted Qwen2.5 sweep also contains two updates with the same 12.3-point target gain that left small-object size accuracy at 23% and 83% (Table C.5). Appendix C varies repair size, label share, learning rate, update type, and field pairs. The original CORD-v2 result provides a case where quantity accuracy largely held (Table F.1).

Comment (2). “little analysis on why different mitigation approaches work or fail”

Response (2). We agree that balancing is well known. We use it as a baseline mitigation to test whether preserving one field also preserves other fields. In Qwen, balancing raised far-depth generation accuracy to 93.0–95.4%, while the all-field check found that tie accuracy fell from 100% to 83.3% (Table 2 and §6.2).

The starting Qwen model already writes depth values outside the schema on 173 of 300 near scenes and 93 of 300 far scenes. The fixed and balanced repairs train the depth tokens directly, and these invalid values disappear in the mixed test. After masking removes that supervision, invalid values rise to 296 on the near half and 288–296 on the far half. Forced choice compares only allowed values, so it cannot reveal these schema errors (Appendix G.1).

To investigate masking, we fine-tuned two SmolVLM2 variants on the same rows with size excluded from the loss in both. Adding material supervision made 13 previously incorrect size answers on large objects correct in free generation across three seeds, with two changes in the opposite direction. Forced choice with the correct prefix still got 12 of those 13 cases wrong. Constraining generation to the two allowed sizes left those predictions unchanged. The predicted improvement on small objects was inconsistent (Appendix G.2).

All 500 responses per seed in the original Qwen masking runs parsed as JSON, but 92.4–96.6% of generated far-depth values were outside the allowed set (Table B.5). In LLaVA-NeXT, a logit offset chosen on 12 separate calibration scenes improved metal accuracy and lowered rubber accuracy in all three runs, with target accuracy unchanged (Appendix G.4). A bias refit in a smaller model likewise recovered only part of the lost accuracy after strong repairs and lowered accuracy on the value used in training (Appendix B.5).

We hope the new control clarifies how training on one field can change the generated answer in another, even when that field is masked. After mitigation, we recommend checking the generated answers for each relevant field and value against the same starting model.

Best,
The Authors

Reviewer_Hfv5OpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Targeted fine-tuning is often used to repair a model’s failure on a narrow problem, while validation typically uses data with the same factor values and fields as the repair data. This paper shows that such practice can mask performance regressions in other fields of a structured output. When a repair specification fixes an input factor at a given value, a non-target output field may receive the same label on every repair example, whereas matched validation never tests the other factor values relevant to that field. On CLEVR-Fields, a Qwen3.5-9B repair raises the accuracy of such a field on repair-matched inputs from 83.7% to 100%, while driving its accuracy on alternate inputs from 86.0% down to 0%. The paper further find that the common practice of removing the affected field from the fine-tuning loss can pass selective preservation checks; yet when the model is required to generate the complete output, accuracy on that field for alternate inputs is only 3.4%–7.6%, with most errors coming from values outside the permitted schema. The paper introduce DoseLint, which uses logged repair factors, reference data, and output dependencies to request preservation tests on the missing input comparisons, in the model’s intended output mode. Given the affected field, 59-row tests on the omitted factor values detect 98.6% of its regressions, compared with 30.3% for tests sampled from the original data distribution. Authors argue that adding alternate examples, as well as mitigation choices such as loss masking, cannot substitute for a preservation test. Corrective updates should be evaluated on the omitted inputs, across affected fields, and in the output mode in which the model is actually used. When repair factors, reference support, and output dependencies are available, they can be used to derive these tests and surface the missing evidence before approval.

Critical strengths
  • The paper investigates performance regressions of targeted fine-tuning on non-target fields, and shows that the common mitigation of removing the affected field from the fine-tuning loss still fails to prevent such regressions under non-selective output generation.
  • It proposes enriching preservation tests with repair factors, reference data, and output dependencies as a way to detect these regressions.
Critical weaknesses
  • The investigation covers only a single model and a single task, so it is unclear how the findings generalize to broader task and model choices.
  • DoseLint is proposed as a way to strengthen preservation testing, but the paper gives almost no concrete description of its method design.
Questions for authors
  • What is the concrete method design of DoseLint? Could you provide a more detailed pipeline description?
  • Does the claim that targeted fine-tuning causes performance regressions on non-target fields also hold for other tasks and models?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Hfv5

Dear Reviewer Hfv5,

Thank you for highlighting the masking result and for asking us to clarify the experimental scope and the DoseLint procedure.

Comment (1). “The investigation covers only a single model and a single task”

Response (1). In addition to the two Qwen models and two tasks in the submission, we added CLEVR experiments with SmolVLM2 and LLaVA-NeXT, scoring the generated answers. After a repair using only large-object examples, SmolVLM2 went from 35 to 26 correct size answers out of 40 on small objects. The same examples changed from correct to incorrect or vice versa in all three seeds, so we count this as one result (Appendix G.2). In LLaVA-NeXT, accuracy on the omitted material fell to zero in all three seeds for both the all-metal and all-rubber repairs (Table 3).

In the submitted Qwen3.5-9B experiments, both depth and quantity scored 100% on inputs matching their respective repair data. On the omitted inputs, CLEVR far-depth accuracy fell to zero, while CORD-v2 quantity accuracy changed from 96% to 94% (Table 3 and Table F.1). The new experiments extend model coverage within CLEVR.

Comment (2). “What is the concrete method design of DoseLint?”

Response (2). Section 6.2 now includes Algorithm 1 and a worked example (Table 5). The depth repair illustrates the six steps.

  1. Read the repair records, reference data, and target field.
  2. Identify concentrated input factors. Depth was near in all 384 repair examples.
  3. Look up alternative values in the reference data. Here, the reference data contain far scenes.
  4. Use declared dependencies to choose non-target fields to check. Here, they include answer_depth. Without a dependency map, consider all non-target fields.
  5. Build a test using labelled examples of the alternative value. If a required test cannot be built, approval remains on hold.
  6. Compare the starting and repaired models on the same examples in the intended output mode. Approval requires a gain on the target and a pass on every requested check.

For full generation, we score the field the model writes and count values outside the allowed set as errors (§4.2). In free generation on 500 far scenes, the fixed repair reduced depth accuracy from 66.4% to 0.0–0.2% (Table 2).

We hope the worked example makes it easier to see what DoseLint checks and how it reaches an approval decision.

Best, The Authors

Reviewer_ntvxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper studies the collateral effects of targeted fine-tuning, where fine-tuning a model to fix one type of problem has unintended effects on other types of problems which could be missed during standard evaluation.
  • It proposes a DoseLint, a tool to identify what samples are not present in the data used to fine-tune to solve the problem and suggests testing on those to evaluate the unintended collateral effects and check if other model capabilities are affected.
  • The papers show this on VQA and receipt extraction tasks with Qwen model where targeted finething improved performance on data belonging to the same distribution as the fine-tuning data but the accuracy on example with a different distribution dropped by a large margin.
  • It evaluates the testing methods by comparing standard tests against the tests suggested by missing samples and find that their tool captures more coverage of collateral effects a The targeted tests detect 98.6% of substantial regressions, compared with 30.3% for standard sampling.
Critical strengths
  • The paper studies an important problem of how targeted finetuning to fix one problem can have unintended and undesirable collateral effects on other samples which are not typically evlauated during such a repair.

  • It proposes a useful tool to which finds gaps in the finetuning data to design tests for evaluation of such collateral effects. The tools could potentially be useful for testing in such settings.

Critical weaknesses
  • The paper has limited novelty. The idea that targeted fine-tuning could have collateral effects is well-known and intuitively expected as shown in prior works on spurious correlations, distribution shifts, catastrophic forgetting, and utility preservation in unlearning.
  • The evaluation in the paper assumes that the colltaterally affected outputs are already known and checks if the tests detects those. However, the method should work on identifying which output fields should be tested, and it is not evaluated. Most experiments are based on closely related repairs in synthetic CLEVR-based tasks and do not suggest that it will extend to real-world complex settings.
Questions for authors

The main evaluation assumes that the affected output field is already known. How effectively does DoseLint identify which non-target fields require preservation testing without this prior knowledge?

1 comment
AuthorsOpenReview ↗

Response to Reviewer ntvx

Dear Reviewer ntvx,

Thank you for noting the value of testing inputs left out of the repair data and for asking how well DoseLint selects the fields to check.

Comment (1). “The paper has limited novelty.”

Response (1). The result we consider most new concerns how a repair is scored. Prior work documents discrepancies between likelihood-based scores and generated answers (Wang et al., 2024; Molfese et al., 2025). We find that these scores can move in opposite directions across an update. On the same 300 far scenes, Qwen depth masking raised the number of correct forced-choice answers from 258 to 281–284, while the number of correct generated answers fell from 197 to 4–12 (Appendix G.1). The gap also runs the other way. In SmolVLM2, forced choice scored 12 of 13 newly correct generated answers as wrong (Appendix G.2). A test in the wrong output mode can therefore pass a repair that damages generated answers or miss a real improvement.

Our study also examines which inputs are needed to check a repair before approval. When the repair and validation data come from the same specification, the repair record can identify input values that validation leaves out. In the partially damaged group, sampling from the original distribution detected at most 5% of regressions even with 600 rows (§6.1).

Comment (2). “How effectively does DoseLint identify which non-target fields require preservation testing”

Response (2). The 98.6% versus 30.3% result in submitted Table 8 compares input sampling with depth already specified. We made this explicit in revised Table 4. We also tested the field request on a new SmolVLM2 repair, with the selection rule fixed before the repair was run. The table gives detection rates over 1,000 draws of 12 scenes (Appendix G.5).

TestSmall-object size lossLarge-object size lossMetal loss
DoseLint’s requested check80%0%0%
Same scenes, all fields scored80%0%95%
Random reference scenes, scored by value75%65%87%

The answer is mixed. The requested check found the small-object loss in 80% of draws, where scenes like the repair data found it in 0%. It missed the metal loss because the declared dependency map left out material. Without a map the procedure scores all fields, and on the same scenes this caught the metal loss in 95% of draws. Field selection that relies on a declared map is only as good as the map. The large-object loss is on the value the repair data contained. The request does not target that value, and repair-matched scenes caught this loss in 55% of draws. Random reference scenes include both sizes, which is why they caught it more often. On one fixed set of 12 images, the requested check flagged 3 of 15 reference losses and all-field scoring flagged 12.

Comment (3). “synthetic CLEVR-based tasks”

Response (3). The new regression experiments use CLEVR. In the original CORD-v2 receipt task, quantity accuracy changed from 96% to 94% (Table F.1). Our public-template survey measures test coverage. In all 17 concentrated cases with checkable official test data, the training-dominant value accounts for at least 97% of the official test rows (Table E.1).

Your question helped us clarify what the requested checks cover and where scoring additional fields helps.

Best,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Paper Revision

Dear Area Chair and Reviewers,

Thank you for your feedback. Hfv5 and Q3LG highlighted the gap between forced-choice evaluation and full generation, and ntvx noted the value of testing inputs left out of the repair data. We followed up on both points in the revised manuscript.

New experimental results

  • Forced choice can also miss improvements. We excluded size from the training loss in both SmolVLM2 variants and added material supervision to one. This made 13 previously incorrect size answers on large objects correct in free generation across three seeds, while two went the other way. Forced choice with the correct prefix still got 12 of those 13 improved cases wrong (Appendix G.2).
  • The regression occurs in two additional model families. A CLEVR repair using only large-object examples reduced SmolVLM2’s correct size answers on small objects from 35 to 26 out of 40. Separate LLaVA-NeXT repairs using only metal or only rubber drove accuracy on the omitted material to zero in all three runs (Table 3).
  • Testing the inputs left out of the repair reveals hidden losses. Images like those used in the SmolVLM2 repair never revealed the small-object size loss (0% of 12-image draws), while the check on small-object images revealed it in 80%. Random sampling detected it in 75% of draws. Scoring all fields on the same small-object images detected a separate loss on metal objects in 95% of draws (Appendix G.5).

How the revision addresses the reviews

  • Hfv5. The submission covered two Qwen models and two tasks. The new experiments expand model coverage within CLEVR. We moved the DoseLint procedure and its worked example into the main text (§6.2, Algorithm 1 and Table 5).
  • ntvx. Revised Table 4 clarifies that both policies were given the depth field. The new field-selection evaluation uses a rule fixed before running the SmolVLM2 repair and shows where the requested checks add coverage to existing validation.
  • Q3LG. The SmolVLM2 control shows that adding material supervision changes size answers even when size is masked. The LLaVA-NeXT experiments show how outcomes vary across seeds at a 50% material split and how a single logit adjustment trades off accuracy between the two material values (Appendices G.2–G.4).

The central result is that even after adding the missing inputs, the scoring mode can change the preservation verdict. On the same 300 far scenes, masking raised Qwen’s correct forced-choice answers from 258 to 281–284, while correct generated answers fell from 197 to 4–12 (Appendix G.1). Revised Table 2 puts the original Qwen scores from both modes side by side.

We removed two supplementary analyses from the appendix, one on single-output natural language inference and one on a proprietary drawing task.

Best,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.