- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Sparse Transformation enhanced Representation Aggregation (STeRA), a parameter-efficient method for Pareto front learning. Existing approaches such as PaMaL and PaLoRA construct preference-conditioned models through linear interpolation of task-specific parameters or LoRA adapters, which restricts the resulting solutions to their convex hull. The paper argues that the optimal trade-off solution should additionally depend on the local curvature of each task objective.
Starting from local quadratic approximations, the authors derive curvature-weighted aggregation formulas in parameter and representation spaces. Because explicitly calculating task-specific Hessians would be expensive, they establish the existence of a sample- and preference-dependent linear transformation from the linearly aggregated representation to the curvature-weighted target. In practice, this dense transformation is replaced by learned block-diagonal rotations over adjacent representation dimensions, together with diagonal scaling for odd-dimensional outputs. Separate LoRA modules generate the transformation parameters from the input and preference vector. The method is evaluated on classification, regression, and scene-understanding benchmarks and generally improves hypervolume over PaLoRA and other Pareto front learning baselines with a comparable parameter count.
Critical strengths
The paper addresses a meaningful limitation of LoRA-based Pareto front learning: linear preference-weighted interpolation restricts solutions to the convex hull of task-specific parameters and may lack sufficient flexibility to represent nonlinear tradeoffs. STeRA provides an efficient and technically interesting alternative by augmenting linearly aggregated representations with sample- and preference-conditioned sparse rotations and scaling transformations. The method is evaluated across six datasets covering classification, regression, and scene understanding with two to seven tasks, and is compared with a broad set of MTL and PFL baselines, including a higher-rank PaLoRA control with a comparable parameter budget. STeRA consistently improves hypervolume while maintaining competitive parameter efficiency. The paper also includes useful ablations on transformation rank, rotation range, preference adjustment, and the rotation and scaling components.
Critical weaknesses
The main concern is that the experiments validate the usefulness of adding learnable transformations but do not validate the claimed curvature-based mechanism. The rotation angles and scaling factors are learned end to end and are not computed from, supervised by, or compared with Hessian-derived curvature; consequently, the improvements could result simply from additional nonlinear, sample-conditioned capacity. The theoretical existence of a dense transformation between linear and curvature-weighted representations does not establish that the proposed sparse adjacent-dimension rotations approximate that transformation or capture the Pareto manifold’s curvature. The paper provides no empirical analysis connecting learned rotations to curvature, and it does not compare the computational cost of explicit curvature modeling with STeRA, despite computational efficiency being a central motivation. In addition, the preliminary formulation and problem statement are difficult to follow, and a clearer illustration of why convex-hull interpolation fails would substantially improve the presentation.
Questions for authors
-
My main concern is that the paper does not empirically validate the curvature-based motivation. Equation 2 defines curvature through the Hessian of each task loss with respect to the representation. However, this establishes curvature of the local loss surface, not necessarily curvature of the learned representation manifold. The paper assumes that a local quadratic model is meaningful and then shows that the final transformation improves task performance. Can the authors provide evidence that the local quadratic approximation is accurate, that the relevant representation-space Hessians contain meaningful directional structure, and that the learned transformation approximates the curvature-weighted target in Equations 2–3? Final task performance alone appears to support the usefulness of the transformation module, but not its proposed connection to curvature.
-
Is the rotation in Equation 4 calculated from curvature in any way? As currently written, the rotation angles appear to be learned entirely through end-to-end optimization rather than computed from the Hessian or its eigendirections. What mechanism encourages these learned rotations to align with high- or low-curvature directions? Could the Hessian, a diagonal/block-diagonal approximation, or Hessian-vector products be used to calculate, supervise, or regularize the rotations, at least on a smaller benchmark? Otherwise, how can the observed gains be distinguished from the general benefit of adding an input- and preference-conditioned transformation module? A parameter-matched comparison with generic conditional transformations would help clarify this issue.
-
The paper motivates sample-wise, preference-conditioned aggregation as a way to avoid the cost of explicit curvature modeling, but it does not quantify that cost or the resulting computational savings. My understanding is that a full Hessian for a -dimensional parameter vector requires storage, while explicit inversion or eigendecomposition can have up to complexity. Representation-space curvature should be cheaper because its dimension is smaller, but sample-wise and task-specific curvature computations may still be expensive. Can the authors provide a theoretical complexity comparison and empirical measurements of training time, inference time, FLOPs, and peak memory for PaLoRA, STeRA, and a feasible curvature approximation? The current parameter-count comparison does not fully establish the claimed computational advantage.
-
The preliminary material and problem formulation are difficult to follow. In particular, the transition from linear interpolation in parameter space to curvature-aware aggregation in representation space is not clearly established before the method is introduced. I suggest adding a dedicated problem-statement subsection that clearly defines the Pareto set, Pareto front, preference vector, parameter-space interpolation, and representation-space aggregation. A simple two-task example showing how linear parameter interpolation is restricted to a convex hull—and why this can prevent reaching a desirable trade-off solution—would make the motivation and contribution substantially clearer. The paper should also distinguish carefully between curvature of the objective with respect to a representation and geometric curvature of the representation manifold.