acceptodds
Withdrawn from ICLR 2027

Curvature-Motivated Sparse Transformation Enhanced LoRA Representation Aggregation for Pareto Front Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pareto front learning (PFL) is able to capture diverse tradeoffs among conflicting tasks in multi-task learning (MTL). Recent approaches adopt linear interpolation of low-rank adapters (LoRA) to offer a scalable parameterization of the Pareto front. However, these methods typically assume that tradeoff solutions lie within the convex hull of task-specific parameters, thereby potentially overlooking the nonlinear geometry of the Pareto manifold shaped by the local curvatures of the task objectives. Consequently, the limited expressiveness of linear interpolation may cause these methods to fall short of the performance gains expected from increasing the LoRA rank. To address these issues, we propose a curvature-motivated PFL method, termed Sparse Transformation enhanced Representation Aggregation (STeRA) from LoRA Modules. Specifically, given the high cost of curvature modeling in parameter space, we instead derive a sample-wise merging strategy in representation space by minimizing the tradeoff-weighted sum of local quadratic objective approximations, yielding a curvature-weighted aggregation of task-specific representations. To avoid materializing multiple task-specific curvature matrices, we establish, under mild conditions, the existence of a transformation from the linearly interpolated representations to the derived curvature-aware representation. Rather than explicitly constructing this dense transformation, we approximate its effect with a learnable sparse transformation that performs independent planar rotations over adjacent representation dimensions, parameterized by sample and tradeoff conditioned rotation angles. By allocating a portion of the LoRA rank budget to modeling these angles, STeRA provides the flexibility to model the underlying nonlinear geometry beyond linear LoRA interpolation. Extensive evaluations across diverse MTL datasets demonstrate the superiority of our method in both performance and parameter efficiency.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Pareto front learning, multi-task learning, LoRA

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_q3WuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Sparse Transformation enhanced Representation Aggregation (STeRA), a parameter-efficient method for Pareto front learning. Existing approaches such as PaMaL and PaLoRA construct preference-conditioned models through linear interpolation of task-specific parameters or LoRA adapters, which restricts the resulting solutions to their convex hull. The paper argues that the optimal trade-off solution should additionally depend on the local curvature of each task objective.

Starting from local quadratic approximations, the authors derive curvature-weighted aggregation formulas in parameter and representation spaces. Because explicitly calculating task-specific Hessians would be expensive, they establish the existence of a sample- and preference-dependent linear transformation from the linearly aggregated representation to the curvature-weighted target. In practice, this dense transformation is replaced by learned block-diagonal rotations over adjacent representation dimensions, together with diagonal scaling for odd-dimensional outputs. Separate LoRA modules generate the transformation parameters from the input and preference vector. The method is evaluated on classification, regression, and scene-understanding benchmarks and generally improves hypervolume over PaLoRA and other Pareto front learning baselines with a comparable parameter count.

Critical strengths

The paper addresses a meaningful limitation of LoRA-based Pareto front learning: linear preference-weighted interpolation restricts solutions to the convex hull of task-specific parameters and may lack sufficient flexibility to represent nonlinear tradeoffs. STeRA provides an efficient and technically interesting alternative by augmenting linearly aggregated representations with sample- and preference-conditioned sparse rotations and scaling transformations. The method is evaluated across six datasets covering classification, regression, and scene understanding with two to seven tasks, and is compared with a broad set of MTL and PFL baselines, including a higher-rank PaLoRA control with a comparable parameter budget. STeRA consistently improves hypervolume while maintaining competitive parameter efficiency. The paper also includes useful ablations on transformation rank, rotation range, preference adjustment, and the rotation and scaling components.

Critical weaknesses

The main concern is that the experiments validate the usefulness of adding learnable transformations but do not validate the claimed curvature-based mechanism. The rotation angles and scaling factors are learned end to end and are not computed from, supervised by, or compared with Hessian-derived curvature; consequently, the improvements could result simply from additional nonlinear, sample-conditioned capacity. The theoretical existence of a dense transformation between linear and curvature-weighted representations does not establish that the proposed sparse adjacent-dimension rotations approximate that transformation or capture the Pareto manifold’s curvature. The paper provides no empirical analysis connecting learned rotations to curvature, and it does not compare the computational cost of explicit curvature modeling with STeRA, despite computational efficiency being a central motivation. In addition, the preliminary formulation and problem statement are difficult to follow, and a clearer illustration of why convex-hull interpolation fails would substantially improve the presentation.

Questions for authors
  1. My main concern is that the paper does not empirically validate the curvature-based motivation. Equation 2 defines curvature through the Hessian of each task loss with respect to the representation. However, this establishes curvature of the local loss surface, not necessarily curvature of the learned representation manifold. The paper assumes that a local quadratic model is meaningful and then shows that the final transformation improves task performance. Can the authors provide evidence that the local quadratic approximation is accurate, that the relevant representation-space Hessians contain meaningful directional structure, and that the learned transformation approximates the curvature-weighted target in Equations 2–3? Final task performance alone appears to support the usefulness of the transformation module, but not its proposed connection to curvature.

  2. Is the rotation in Equation 4 calculated from curvature in any way? As currently written, the rotation angles appear to be learned entirely through end-to-end optimization rather than computed from the Hessian or its eigendirections. What mechanism encourages these learned rotations to align with high- or low-curvature directions? Could the Hessian, a diagonal/block-diagonal approximation, or Hessian-vector products be used to calculate, supervise, or regularize the rotations, at least on a smaller benchmark? Otherwise, how can the observed gains be distinguished from the general benefit of adding an input- and preference-conditioned transformation module? A parameter-matched comparison with generic conditional transformations would help clarify this issue.

  3. The paper motivates sample-wise, preference-conditioned aggregation as a way to avoid the cost of explicit curvature modeling, but it does not quantify that cost or the resulting computational savings. My understanding is that a full Hessian for a PP-dimensional parameter vector requires O(P2)O(P^2) storage, while explicit inversion or eigendecomposition can have up to O(P3)O(P^3) complexity. Representation-space curvature should be cheaper because its dimension is smaller, but sample-wise and task-specific curvature computations may still be expensive. Can the authors provide a theoretical complexity comparison and empirical measurements of training time, inference time, FLOPs, and peak memory for PaLoRA, STeRA, and a feasible curvature approximation? The current parameter-count comparison does not fully establish the claimed computational advantage.

  4. The preliminary material and problem formulation are difficult to follow. In particular, the transition from linear interpolation in parameter space to curvature-aware aggregation in representation space is not clearly established before the method is introduced. I suggest adding a dedicated problem-statement subsection that clearly defines the Pareto set, Pareto front, preference vector, parameter-space interpolation, and representation-space aggregation. A simple two-task example showing how linear parameter interpolation is restricted to a convex hull—and why this can prevent reaching a desirable trade-off solution—would make the motivation and contribution substantially clearer. The paper should also distinguish carefully between curvature of the objective with respect to a representation and geometric curvature of the representation manifold.

Reviewer_qQANOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of Pareto front learning (PFL). It argues that the manifold-based PFL should involve the consideration of objectives' curvature rather than merely linearly combining them. It later turned to the representation-level hessian for efficiency purposes and derived the closed-form optimal representation, which contains both the weighting and transformation terms. Motivated by this, it further proposed a learnable sparse transformation enhanced representation aggregation module to implement above two terms. Extensive experiments across multiple MTL benchamrks have demonstrated the effectiveness of the proposed solution.

Critical strengths
  1. The intuition is straightforward, and therefore the whole paper is generally easy to follow.

  2. The results appear to be competitive across multiple mainstream benchmarks.

  3. The proposed learnable solution is well motivated and appears to be interesting.

Critical weaknesses
  1. Opt to representation-level hessian rather than parameter-level hessian. Although more efficient, I am not sure if it's applicable since parameter features are neither equal nor one-to-one corresponding due to symmetry structure in NN.

  2. The design in Figure 2 should be more correlated with Eqn. (3). In my understanding, the upper branch corresponds to the weighting attribute, while the lower branch corresponds to the non-linear ones. If my understanding is right, please indicate it more explicitly in the paper.

  3. If Theorem 1 is just to prove the existence of nonlinear transformation in optimal z*, then I didn't see the necessity of Theorem 1, since Eqn. (3) has already motivated the subsequent designs.

  4. Computational comparison is expected, since one of motivations of this paper is the existing solutions' heavy computation. Although some qualitative analysis has been provided, quantitative ones are also expected.

  5. Additional introduced parameters should also be quantified for fair comparison.

Questions for authors

Please focus on addressing my raised weaknesses. If my concerns were well addressed, I am pleased to further increase the rating.

Reviewer_cdVJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper the authors propose a curvature-motivated Pareto Front Learning method namely Sparse Transformation enhanced Representation Aggregation (STeRA) from LoRA Modules. To circumnavigate the high cost of curvature modeling in parameter space the authors instead derive a sample-wise merging strategy in representation space. Beyond simpler linear formulation STeRA provides the flexibility to model the underlying nonlinear geometry. The proposed approach achieves competitive performance across different multi-task learning benchmarks.

Critical strengths

Paper approaches a critical challenge of multi-task learning and questions if simple linear interpolation of low-rank adapters (LoRA) is enough to model complex pareto fronts. The curvature based theoretical intuition highlighted in the paper is interesting and is relevant for the broader community.

Critical weaknesses
  1. While the proposed solution focus on curvature, the chain of simplifications needed for tractability (block-diagonal layer-wise Hessians, etc.) progressively removes the dependence on curvature. It is unclear whether the gains come from the proposed mechanism or simply from added conditional capacity. When compared to existing baselines many margin is relatively small.

  2. The models used in the experiments are all small scales. The proposed method should be demonstrated on larger pretrained transformers.

  3. The computational cost and wall clock latency of the proposed approach is not elaborated in the paper.

Questions for authors
  1. Can the authors demonstrate their claim on a smaller 2-D or 3-D toy problem where the true front is known. They can highlight on that specific problem how the proposed method is better than PaLoRA.

  2. Another direction will be actually working with a much bigger pretrained model architecture to demonstrate the capability of the approach at scale.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.