acceptodds
Under review as a conference paper at ICLR 2027

Beyond Training Support: Parameterization Selects Fine-Tuning Endpoints

Abstract

Fine-tuning on a narrow support often does not determine a unique model function. Many updates can match all training predictions yet behave differently outside the observed support. Optimization therefore does more than fit the data: it also selects how the model extends beyond the training set. We formalize this choice through a parameterization-conditioned endpoint selector. Specifically, for linear adaptation coordinates, the selected endpoint is an explicit metric-minimum right inverse. For balanced signed-square coordinates, it is a hypentropy-minimum interpolant, with initialization scale controlling a transition between sparse and Euclidean regimes. We then construct parameterizations that produce identical training-prediction trajectories at every step but converge to different endpoints. Next, we derive an analytic boundary for when their held-out ordering reverses. Conversely, we characterize when two selectors agree on a fixed held-out witness and show that, once the training support identifies that witness, all interpolating endpoints must agree. Finally, experiments in solvable systems, frozen representations up to 30B parameters, and matched-fit Transformer adaptations separate endpoint selection from expressivity, convergence, and learning amount. Together, the theory and experiments explain why the same narrow-domain fit can lead to different functions, and when additional support removes that freedom.

Primary area:
optimization
Keywords:
fine-tuning generalization, parameterization, endpoint selection, implicit bias, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_cT4HOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an "endpoint-selection framework" to analyze fine-tuning. It argues that narrow support fails to uniquely determine a model, and different parameterizations select different interpolating endpoints. The authors provide theoretical derivations for linear and signed-square factorized coordinates, and conduct empirical experiments

Critical strengths

None

Critical weaknesses
  1. Unacceptable writing quality There are too many jargons such as "training fiber" "endpoint selector", "witness-specific equivalence", and these notions are trivially linear algebra items. Besides, all the theorems look separated, I do not understand how they are connected to a coherent story. The paper's presentation is disproportionately complicated relative to its underlying mathematical content.

  2. The theoretical contributions are trivial and heavily overclaimed.

The core theoretical result (e.g., Theorem 6.3, "support-identifiability collapse") is merely the fundamental theorem of linear algebra (row-space/null-space decomposition). Standard minimum-norm interpolants and linear right-inverses are repackaged with new terminology ("parameterization-conditioned endpoint selection"). Claiming this as a novel, profound theory is unacceptable.

Questions for authors
  1. How do these theorems connected to each other and your empirical observations?
  2. What is the main nontrivial theoretical contribution beyond existing implicit-bias results, and which experiment actually validates that contribution in a nonlinear Transformer?
Reviewer_ocEnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how different fine-tuning parameterizations can select different solutions despite similar training predictions. It establishes different behavior outside the training support.

Critical strengths

This paper studies a meaningful question concerning the implicit bias of fine-tuning and its consequences for out-of-support generalization. The theoretical framework is quite coherent.

Critical weaknesses

My main concern is the insufficiently organized experimental evidence. Despite claiming extensive evaluations, the paper provides no experimental results tables and relies on a few figures and selectively reported results, with much of the essential evidence deferred to the appendix.

The theoretical novelty is limited relative to established implicit-bias results, and the connection between the theoretical selectors and practical LoRA or DoRA adaptations is not convincingly demonstrated.

Questions for authors
  1. Can the authors present a comprehensive comparison of the main experimental results, including training-fit discrepancies and held-out behavior across models and adaptation methods, directly in the main paper?

  2. What new theoretical or empirical insight does the framework establish beyond existing implicit-bias theory, and how can endpoint selection be distinguished from other optimization effects in the actual Transformer experiments?

Reviewer_23QsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies how parameterization affects the functions selected by fine-tuning when the training data do not uniquely determine the model. Endpoint-selection rules for linear adaptation coordinates and balanced signed-square parameterizations are presented, and the paper constructs examples where identical training-prediction trajectories lead to different endpoints, and characterizes when held-out predictions must agree. Experiments consider solvable systems to frozen representations and Transformer fine-tuning. The main claim is that parameterization can affect off-support behavior even when training fit is held fixed.

Critical strengths

The paper makes the distinction between fitting the observed support and selecting a particular extension beyond it. The construction showing that identical training-prediction trajectories can lead to different endpoints is a clean way to illustrate this distinction.

The paper also attempts to validate its claims beyond toy linear systems, including experiments with large pretrained representations and Transformer adaptations.

Critical weaknesses

Much of the theoretical contribution is a repackaging of standard linear algebra and implicit-bias results. For example, the characterization of when a linear functional is constant over the interpolation set follows directly from the null-space/range-space relationship. The norm-minimum interpolant is standard.

The experiments do not yet convince me that the framework provides a substantial new explanation of practical fine-tuning behavior. Overall, the paper makes a coherent point, but the novelty and empirical significance seem limited.

Questions for authors

What are the precise theoretical results that are new relative to existing work on minimum-norm interpolation, implicit bias, and hypentropy regularization? Please identify the closest prior results and explain the technical difference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.