acceptodds
Under review as a conference paper at ICLR 2027

Training a Few Anchors for a Dense Update

Abstract

Parameter-efficient fine-tuning reduces the number of optimized variables, but most methods impose a parameterization without considering the geometry of the downstream objective. We GATA: Geometry Aware Task Alignment, which selects a task-conditioned adaptation space and represents it using physical parameter anchors. Candidate Hessian modes are ranked by their predicted improvement under a damped quadratic model, . Q-DEIM then selects parameter coordinates, and a fixed affine map reconstructs their updates over all adapted parameters. AdamW therefore optimizes only anchor variables while producing a dense update. Across five-seed classification-head experiments, task alignment consistently outperforms curvature-only and dimension-matched random spaces. On ViT-B/16/CIFAR-10, it improves accuracy at from for leading-curvature modes to , while its Q-DEIM representation retains . At , task-aligned Q-DEIM reaches approximately accuracy using 150 optimized variables, comparable to rank-4 LoRA using 3,112. On ResNet-18/CIFAR-10, the task-aligned space reaches at , compared with for a matched random space. Complete Hessian spectra further show that curvature rank grows with model size before saturating at a task-dependent level, while GPT MLP measurements exhibit sublinear growth of spectral-energy rank. These results support curvature dimension as an interpretable adaptation budget and physical anchors as a compact interface for optimization and communication.

Primary area:
learning theory
Keywords:
Parameter Efficient Fine-Tuning, Hessian geometry, Q-DEIM, low-dimensional optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7xFMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GATA (Geometry Aware Task Alignment), which constructs a task-conditioned adaptation subspace and represents its dense updates through a small number of physical parameter coordinates. Candidate Hessian eigenmodes are estimated using Hessian–vector products and Lanczos iteration, then ranked by their predicted improvement under a damped quadratic model, (vi⊤g)2/(λi+γ)(v_i^\top g)^2/(\lambda_i+\gamma). Q-DEIM selects kkphysical anchors, and a fixed reconstruction map propagates their updates across all NN adapted parameters. The paper provides an exact interpolation and span-preservation argument, together with a separate local affine-force approximation bound.

The main evaluation consists of five-seed CIFAR-10 classification-head experiments with frozen ResNet-18 and ViT-B/16 feature extractors. Task-guided selection substantially outperforms curvature-only and random subspaces at small adaptation budgets. Further experiments compare classification-head versions of PEFT methods, validate Lanczos against explicit Hessian eigendecomposition on a small MLP, and examine curvature-dimension scaling using MNIST classifiers and GPT MLP blocks. The results support the usefulness of task-conditioned subspace selection in the controlled head-adaptation setting. The broader scaling and practical-efficiency conclusions require stronger evidence.

Critical strengths
  1. The controlled experiments provide convincing evidence that task information matters when selecting a small adaptation subspace. The strongest empirical contribution is the comparison between task-guided, curvature-only, and dimension-matched random spaces. The primary experiments use matched features, initializations, data splits, and optimization budgets, with development-based learning-rate selection and reporting across five seeds. On ViT-B/16 at k=20k=20, task-aligned selection achieves 83±1.8%83\pm1.8\%, compared with 30±6.6%30\pm6.6\%for curvature-only selection and 21±4.8%21\pm4.8\% for a random space. This large difference supports the central observation that high curvature alone is not a sufficient criterion for useful adaptation directions. It is a substantive result within the evaluated setting.

  2. The paper clearly separates subspace selection from coordinate representation and supports the exact construction with relevant checks. For an invertible selected-coordinate matrix, the reconstruction basis satisfies both PS⊤B=IkP_S^\top B=I_k and span⁡(B)=span⁡(Vk)\operatorname{span}(B)=\operatorname{span}(V_k). Thus, the optimized variables genuinely correspond to updates at selected physical coordinates, while the resulting parameter update remains dense. The paper also acknowledges that this representation still requires gradient information from unselected coordinates and can produce different optimization trajectories from orthonormal subspace coordinates. These distinctions make the underlying construction technically transparent. The exact-versus-Lanczos validation on a small whole-network Hessian provides additional evidence that the numerical eigensolver can reproduce the intended construction in that controlled case.

Critical weaknesses

1. The curvature-dimension scaling conclusion is not adequately substantiated by the reported measurements, and the associated mathematical interpretation contains inaccuracies.

In the GPT experiment, Appendix D.6 extracts at most 1,000 Ritz values, removes negative entries, and defines the reported dimension as normalized against the sum of the extracted positive spectrum. While this limitation is disclosed in the appendix, Section 6 interprets the resulting curves directly as evidence that the underlying dominant curvature dimension grows sublinearly and saturates. This inference cannot be justified by the truncated normalization alone.

A simple counterexample illustrates this flaw. Applying the same normalization to a full-rank matrix H=INH=I_N (with N≥1000N\geq1000) using 1,000 extracted eigenvalues yields a fixed k95=950k_{95}=950 for any NN, even though the Hessian is full rank and exhibits no spectral concentration. This means the observed saturation pattern can arise from the extraction budget cap, rather than the intrinsic geometry of the loss landscape. Moreover, spectral energy in Appendix B.1 is defined using squared eigenvalues and an independently estimated tr⁡(H2)\operatorname{tr}(H^2), which is a distinct quantity from the truncated positive-eigenvalue sum used in the GPT experiment. The relationship between these two metrics, and the validity of conclusions drawn from each, needs to be clarified.

For the MNIST experiment, even though the full spectrum is computed, its interpretation is also constrained by the fixed calibration sample size �0��0�. The reported rank bound rank⁡(H)≤min⁡{(C−1)w,  M(C−1)}\operatorname{rank}(H)\leq\min\{(C-1)w,\;M(C-1)\} inherently introduces a sample-dependent plateau. The existing experiments thus fail to disentangle a task-intrinsic dimensional limit from the ceiling imposed by finite calibration samples.

Finally, Section 8 concludes from low Hessian rank that not all parameters are linearly independent. This deduction is mathematically unsound. Hessian rank deficiency describes local second-order variation in the loss function, which is conceptually distinct from linear dependence among the parameter coordinates themselves. Given that characterizing curvature-dimension scaling is framed as a core contribution, these issues carry significant weight for the paper’s central claims.


2. The PEFT experiments do not yet establish a reliable practical advantage for the proposed method.

The core adaptation experiments only train randomly initialized linear classification heads — with just 5,120 or 7,680 parameters — on cached, frozen features. The PEFT baselines also use single-head variants, including an explicitly simplified AdaLoRA-style control. While this setup is useful for comparing constrained parameterization schemes, it provides limited insight into how the method performs when adapting internal layers of pretrained networks. The broader framing of the method as an alternative to dense fine-tuning and low-rank adapters is therefore not yet supported by experiments at relevant model scales.

Furthermore, reducing the number of optimized scalars does not necessarily reduce total adaptation cost. The paper correctly discloses �1��1� reconstruction storage. For �2��2� and �3��3�, the explicitly stored mapping contains 1,152,000 scalars, substantially exceeding the parameter count of the entire adapted head. While Table 10 acknowledges large reconstruction overheads, it does not provide matched end-to-end comparisons of accuracy, runtime, and peak memory against the main baselines. The optimized-variable axis in Figure 2 therefore only demonstrates a compact optimization state, not an overall memory or computational advantage.

There is also an unresolved mismatch between the stated method and its experimental implementation. Section 4 describes exact Q-DEIM, but the PEFT configuration uses a pseudoinverse cutoff of 10−210^{-2} and clips mapping values to [−0.5,0.5][-0.5, 0.5]. Appendix B.6 explicitly notes that such stabilization generally changes the selected span. It is therefore necessary to clarify which results use exact interpolation and which use the stabilized implementation, before attributing the reported performance to an exactly preserved task-selected subspace.

Questions for authors

Q1. Interpretation and validation of the scaling claim — clarification and targeted validation. How can the GPT experiment distinguish genuine curvature-dimension saturation from the fixed 1,000-Ritz-value extraction limit? Please report sensitivity to the extraction budget and, where feasible, use an independently estimated spectral denominator. Please also reconcile the GPT metric in Appendix D.6 with the squared-curvature energy definition in Appendix B.1 and the description of “structural ranks” in Section 4. For MNIST, does the saturation pattern persist when the number of calibration examples varies? This would help separate task-dependent geometry from the known M(C−1)M(C-1) finite-sample ceiling.

Q2. Exact versus stabilized reconstruction — clarification and correction. Which figures and tables use the exact inverse, and which use the truncated pseudoinverse and clipping specified in the configuration tables? Were these stabilization settings actually enabled in the reported PEFT results? For stabilized runs, please provide the principal-angle distance from the originally selected subspace, the relevant conditioning measurements, and a comparison with exact reconstruction. Appendix B.6 correctly notes that stabilization can change the span, so these results should be distinguished from experiments covered by the exact span-preservation argument.

Q3. End-to-end adaptation cost — targeted validation of the efficiency claim. What are the total calibration and training times, peak memory, and stored adaptation-state sizes for GATA, dense head tuning, and LoRA at comparable accuracy on the same hardware? Please include Hessian–vector products, eigensolver and QR computation, basis or map storage, reconstruction, and gradient computation. Please also explain how the FLOPs ratios in Table 10 were obtained and whether they are measured or estimated. Under this complete accounting, which efficiency advantages are demonstrated, and which are limited to optimizer-state size or communication after a shared reconstruction map has already been distributed?

Q4. Contribution of task scoring and physical anchors — targeted attribution of the reported gains. Figure 6 shows very similar performance for gradient-aligned and task-aligned mode selection. With the same candidate pool and paired seeds, what is the incremental benefit of the curvature-normalization factor 1/(λi+γ)1/(\lambda_i+\gamma)? Please provide paired accuracy differences and their variability rather than relying only on the overlapping curves. For a fixed selected basis VkV_k, what measurable benefit does Q-DEIM provide over directly optimizing kk coefficients in that same basis? Both representations use kk optimized scalars. Distinguishing any conditioning or practical-interface benefit from the benefit of selecting a task-relevant subspace would clarify the contribution of physical anchors.

Reviewer_AXUtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GATA (Geometry Aware Task Alignment), a parameter-efficient fine-tuning method that selects task-relevant Hessian directions using gradient information and predicted loss reduction. Q-DEIM selects a small number of physical parameter anchors to reconstruct dense updates through a fixed linear mapping. Experiments on CIFAR-10 with frozen ResNet-18 and ViT-B/16 backbones demonstrate promising performance with few optimized variables. The paper also analyzes Hessian rank scaling across model sizes and tasks.

Critical strengths
  1. The paper presents a clear and well-motivated approach that combines Hessian geometry and task gradient information to select adaptation directions. The experimental results demonstrate promising performance under limited optimization-variable budgets.

  2. The Q-DEIM-based parameterization provides an interesting way to reconstruct dense parameter updates from a small number of physical anchors. The paper also includes theoretical analysis and controlled experiments to support the proposed framework.

Critical weaknesses
  1. GATA shows limited independent improvement over Gradient-aligned Selection. In Figure 6, the accuracy curves of Gradient-aligned Selection and GATA are very close, especially when k > 20. Compared with the substantial improvement over the Curvature-only method, the performance gains may mainly come from incorporating task gradient information rather than the proposed Hessian curvature-weighted scoring function.

  2. Although GATA optimizes only k variables, it requires a reconstruction matrix B of size N x k. This means that the storage requirement of the reconstruction matrix is over a hundred times that of the original classification head parameters.

  3. The main experiments only train the final linear classification head while keeping the ResNet-18 and ViT-B/16 feature extractors frozen. This differs significantly from actual multi-layer Transformer fine-tuning. The paper lacks comparisons with LoRA and other methods in terms of performance and efficiency on real multi-layer Transformer fine-tuning tasks.

  4. Q-DEIM and Direct Subspace have the same expressive space, and the independent advantage of Q-DEIM remains unclear.

Questions for authors

Please refer to the Critical Weaknesses.

Reviewer_4Y3wOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose GATA, a method for fine-tuning a model inside small subspace picked from the local geometry of the downstream loss. Lanczos gives Hessian eigenvectors at the starting point, and these are ranked by the loss decrease that a damped quadratic model predicts along each of them. Q-DEIM then maps the top modes to the same number of physical weights, called anchors. Only the anchors are trained with AdamW. A fixed linear map spreads their change over all other weights. The main experiments train a linear classification head on frozen ResNet-18 or ViT-B/16 features for CIFAR-10. There GATA is compared with curvature-only, gradient-only and random subspaces of the same size, and also with dense training and LoRA-style adapters on the same head. The paper further measures how Hessian rank grows with width and number of classes, and proves a one-step error bound for the linear approximation behind the method

Critical strengths
  1. On ResNet-18 and ViT-B/16, the proposed method shows a clear advantage at a small anchor budget.
  2. The ablation is well designed. The comparison with curvature-only and gradient-only selection separates the contribution of low dimensionality and of gradient-guided direction selection from the authors’ overall contribution.
  3. The experimental protocol is described in detail, and I did not find incorrect points in it.
  4. The theoretical part is correct. I checked it and found no errors.
Critical weaknesses
  1. If I understand correctly, compared with dense fine-tuning the proposed method saves only optimizer states, not computation time or memory. The need to store the matrix BB raises a separate question. It has size N⋅kN \cdot k, which is quite large.
  2. The method requires computing the gradient with respect to all parameters, unlike LoRA-style adapters. This calls the computational efficiency of the proposed method into question.
  3. The paper does not study the scalability of the proposed method. Given the previous weaknesses, this is especially relevant.
  4. The authors adapt only the model heads. The paper does not study whether the method applies to layers other than the late ones, which matters, for example, for language tasks
Questions for authors

Can authors provide experiments on more complex fine-tuning problems? For example, fine-tuning OPT-1.3B on SST-2 with a comparison against competitors in quality, time and memory. This would address most of my concerns

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.