Training a Few Anchors for a Dense Update

Parameter-efficient fine-tuning reduces the number of optimized variables, but most methods impose a parameterization without considering the geometry of the downstream objective. We GATA: Geometry Aware Task Alignment, which selects a task-conditioned adaptation space and represents it using physical parameter anchors. Candidate Hessian modes are ranked by their predicted improvement under a damped quadratic model, $({\mathbf{v}}_i^{\top }\mathbf{g}{)}^2/({\lambda }_i+\gamma )$. Q-DEIM then selects $k$ parameter coordinates, and a fixed affine map reconstructs their updates over all $N$ adapted parameters. AdamW therefore optimizes only $k$ anchor variables while producing a dense update. Across five-seed classification-head experiments, task alignment consistently outperforms curvature-only and dimension-matched random spaces. On ViT-B/16/CIFAR-10, it improves accuracy at $k=20$ from $30\pm 6.6\%$ for leading-curvature modes to $83\pm 1.8\%$, while its Q-DEIM representation retains $84\pm 1.8\%$. At $k=150$, task-aligned Q-DEIM reaches approximately $92\%$ accuracy using 150 optimized variables, comparable to rank-4 LoRA using 3,112. On ResNet-18/CIFAR-10, the task-aligned space reaches $78.65\pm 0.85\%$ at $k=250$, compared with $60.66\pm 1.45\%$ for a matched random space. Complete Hessian spectra further show that curvature rank grows with model size before saturating at a task-dependent level, while GPT MLP measurements exhibit sublinear growth of spectral-energy rank. These results support curvature dimension as an interpretable adaptation budget and physical anchors as a compact interface for optimization and communication.