Curvature-Motivated Sparse Transformation Enhanced LoRA Representation Aggregation for Pareto Front Learning

Pareto front learning (PFL) is able to capture diverse tradeoffs among conflicting tasks in multi-task learning (MTL). Recent approaches adopt linear interpolation of low-rank adapters (LoRA) to offer a scalable parameterization of the Pareto front. However, these methods typically assume that tradeoff solutions lie within the convex hull of task-specific parameters, thereby potentially overlooking the nonlinear geometry of the Pareto manifold shaped by the local curvatures of the task objectives. Consequently, the limited expressiveness of linear interpolation may cause these methods to fall short of the performance gains expected from increasing the LoRA rank. To address these issues, we propose a curvature-motivated PFL method, termed Sparse Transformation enhanced Representation Aggregation (STeRA) from LoRA Modules. Specifically, given the high cost of curvature modeling in parameter space, we instead derive a sample-wise merging strategy in representation space by minimizing the tradeoff-weighted sum of local quadratic objective approximations, yielding a curvature-weighted aggregation of task-specific representations. To avoid materializing multiple task-specific curvature matrices, we establish, under mild conditions, the existence of a transformation from the linearly interpolated representations to the derived curvature-aware representation. Rather than explicitly constructing this dense transformation, we approximate its effect with a learnable sparse transformation that performs independent planar rotations over adjacent representation dimensions, parameterized by sample and tradeoff conditioned rotation angles. By allocating a portion of the LoRA rank budget to modeling these angles, STeRA provides the flexibility to model the underlying nonlinear geometry beyond linear LoRA interpolation. Extensive evaluations across diverse MTL datasets demonstrate the superiority of our method in both performance and parameter efficiency.