acceptodds
Under review as a conference paper at ICLR 2027

What Fit Hides: The Geometry of Few-Shot Subpopulation Repair

Abstract

Two model updates can produce nearly identical changes in fixed edit-loss measurements yet generalize differently to the wider target subpopulation. We study this ambiguity in few-shot subpopulation repair, where a model is edited using a small correction set. A local geometric analysis quantifies how much held-out target improvement can remain ambiguous given identical or approximately matching edit measurements. With additional target, non-target, and reference probes, we construct a diagnostic update that achieves a specified local target improvement while minimizing a regularized non-target representation drift cost within the edit-gradient span. The same geometry predicts its locality advantage over the projected target-gradient update. Across six CIFAR-100 model runs, edit-loss differences shrink approximately quadratically with update size, while held-out target-score differences shrink approximately linearly. Using the same probes and matching validation improvement, the diagnostic achieves 16.1–18.5% higher model-median test target gain per unit off-target drift than the projected target gradient. Validation geometry predicts the projected-gradient-to-diagnostic test drift ratio without fitted calibration, with 4.31% median error relative to the excess ratio above one, compared with 22.04% for a leave-one-model-out constant predictor. The locality advantage remains with fewer probes and an additional edit-loss descent constraint. Paired continuations show that updates with the same immediate target improvement can have different effects after training resumes. Waterbirds and controlled ImageNet-20 reproduce the generalization and locality distinctions under a natural subgroup failure and with larger ResNet and ViT models.

Primary area:
learning theory
Keywords:
Model Editing, Representation Learning, Model Repair, Generalization, Continual Learning, Local Geometry, Machine Learning Theory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UzDsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a regularized minimum-drift diagnostic update for few-shot subpopulation repair. Specifically, it argues that edit-set fit does not establish held-out target improvement or non-target preservation. To address this, the diagnostic uses additional target and non-target probes to achieve a specified first-order target improvement while minimizing regularized non-target representation drift within the edit-gradient span. Experimental results demonstrate that the diagnostic achieves higher held-out target gain per unit off-target drift than the projected target gradient and the diagonal ablation on CIFAR-100.

Critical strengths
  1. The paper breaks down few-shot subpopulation repair into distinct objectives, making the research problem clearer.

  2. The related work provides a detailed discussion of existing approaches and their limitations, helping contextualize the proposed analysis.

  3. The paper provides a local geometric analysis that offers theoretical guidance for constructing a new method.

Critical weaknesses
  1. The paper reads as an incomplete study, with an unclear logical structure and exposition that is difficult to follow.

  2. The depth of the theoretical contribution requires stronger justification. Although the authors attempt to use theory to guide method design, the connections between the theoretical results and the specific challenges of few-shot subpopulation repair are not sufficiently explored.

  3. The experimental coverage is insufficient to establish the broader applicability of the findings across diverse repair settings.

  4. This does not affect my recommendation, but the writing gives the impression of heavy LLM assistance without improving clarity or coherence.

Questions for authors
  1. Could the authors clarify the logical connections between the problem formulation, theoretical analysis, and diagnostic update?

  2. What insights beyond standard mathematical results specifically address the challenges of few-shot subpopulation repair?

  3. Can the authors provide additional evidence under natural model degradation and resumed training across diverse settings?

Reviewer_gNHtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies why fitting a small correction set does not necessarily repair the wider target subpopulation. It uses local geometry to characterize target effects that edit measurements leave undetermined and constructs a diagnostic update that minimizes non-target representation drift for a specified target-score improvement.

Experiments test these predictions on CIFAR-100, with additional generalization and locality tests on Waterbirds and ImageNet-20 using ResNet and ViT models. Paired training continuations also show that equal immediate improvements can have different later effects.

Critical strengths

An interesting geometric account with testable predictions. The paper connects ambiguity in edit measurements, target improvement, and non-target drift within one framework. Predicting the test drift ratio from validation geometry without fitted calibration is a meaningful result beyond simply demonstrating that editing can fail.

Careful controls and transparent limitations. The same-information comparisons, separate test probes, reduced-probe experiments, and paired continuations strengthen the evaluation. The paper also acknowledges that its diagnostic is not necessarily a better practical editor and that representation preservation does not guarantee behavioral preservation.

Critical weaknesses

The connection to useful model improvement remains insufficiently established. The main locality result is a 16.1–18.5% improvement in target-margin gain per unit representation drift, measured for small updates in a restricted gradient span. This supports the local geometric claim, but does not establish a comparable improvement in meaningful repair outcomes.

The Waterbirds control makes this distinction particularly important: balanced last-layer retraining achieves 92.8% target and 90.1% non-target accuracy, compared with 82.8% and 88.0% for the diagnostic. The methods use different update spaces, so this does not contradict the theorem. However, it leaves the practical value unresolved: when should the diagnostic be used, and what decision does it improve over simpler repair and validation procedures?

The broader significance of the theoretical findings needs stronger justification. The identifiability results largely follow from null-space relationships, while the minimum-drift update has a standard metric-preconditioned form. Their application is interesting, but the experiments primarily validate deliberately constructed local distinctions. For example, edit-equivalent pairs are constructed to cancel the defining measurements, and independent augmentations do not show the same quadratic contraction.

ResNet and ViT experiments provide some architectural breadth, but the detailed locality and retention evidence remains concentrated in six CIFAR runs sharing an initialization and split. The paper therefore establishes a local diagnostic more convincingly than broadly actionable guidance for repair across models and failure types.

Questions for authors

Practical clarification: What concrete repair decision should change after computing these geometric quantities? Which reported result best demonstrates that benefit beyond validating the local theory?

Behavioral clarification: Does the lower representation drift reliably identify updates with better target accuracy and non-target preservation? How should the Waterbirds last-layer result inform the intended use of the diagnostic?

Scope clarification: Which conclusions are expected to hold beyond the fixed edit measurements and restricted update span, and which are supported empirically for ordinary learned edits?

Reviewer_4HWJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies few-shot subpopulation repair, i.e. updating a model using a small correction/edit set. They look at whether this results in improving performance on other target examples and preserving non-target behavior. Its local geometric analysis shows that matching edit losses do not necessarily correspond to matching target improvement, and constructs an update that minimizes non-target representation drift for a specified target gain. Experiments test these predictions and show that updates with similar immediate benefits can diverge after training resumes.

Critical strengths

The problem that the paper explores is interesting: how repairing capabilities affect performance on other examples, whether they are from the distribution that is being repaired or outside of that distribution.

Critical weaknesses

This paper is very difficult to read. The writing is very hard to follow, and in general, there is very little context about the overarching story, why an experiment is being run, or what the reader should pay attention to. There are numerous references to the appendix that contain important experimental details, with comparatively few references to what's in the main paper. There is one figure in the main text, and it's not properly discussed what it's trying to show. Overall, this paper is not in the state to be accepted, and likely needs a substantial rewrite.

If I understand correctly, Figure 1b is essentially asking whether the ratio of two drift magnitudes predicted from validation Jacobians agrees with the measured ratio on test images. Given the step size is small, it's essentially doing a one step Taylor approximation check, seeing whether validation geometry is close to test geometry. It seems not that surprising that they would be close, so I am not sure why this experiment is significant.

Questions for authors
  • Does the locality advantage persist for updates large enough to produce meaningful accuracy improvements, or across multiple repair steps with geometry recomputed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.