acceptodds
Withdrawn from ICLR 2027

ISC-LoRA: Intrinsic Spectral Continuation for Multitask Dense Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multitask learning with low-rank adaptation (LoRA) efficiently adapts a shared visual model to complementary dense prediction tasks, including semantic segmentation, boundary detection, and geometry estimation. Reweighting the visual features based on their spatial distribution adaptive to the specific task and inputs can further improve these predictions. To do this, multitask LoRA methods often separate low-rank updates into shared and task-private branches. The low-dimensional features in each private branch retain the spatial grid of the current input and can reflect task-specific spatial variation. However, the spatial context does not directly indicate which frequencies should be emphasized or how strongly, limiting the adaptation capability to the specific input or task. Building on this structure, we introduce Intrinsic Spectral Continuation LoRA (ISC-LoRA), a subspace-conditioned training scheme that derives a bounded response over the full spatial-frequency grid from each private branch's current features and subspace. This response is recomputed for each input and private branch, adapting the emphasis of spatial frequencies to each task and image without predefined frequency bands. To make the response intrinsic to this subspace rather than to the particular LoRA rank coordinates used to represent it, our method combines frequency preferences along the principal directions of feature variation using geometry-aware regularization and covariance normalization. We further prove that ISC-LoRA can reweight the spatial frequencies of the private output within a controlled range, with the response determined by the current input and task-private subspace, independent of the chosen LoRA rank basis. Notably, its continuation schedule reduces the modulation strength to zero, making the spectral operator the identity while retaining the learned LoRA factors. The resulting adapters can therefore be evaluated and deployed using standard LoRA computation, without spectral operations. Experiments on PASCAL-Context and NYUD-v2 show that ISC-LoRA outperforms representative multitask adaptation methods. Beyond natural images, evaluations on ten two-dimensional Cell Tracking Challenge (CTC) datasets further demonstrate its advantages over representative adaptation baselines in cell segmentation, detection, and tracking. Our code and checkpoints are available at https://anonymous.4open.science/r/ISC-LoRA.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multi-task learning, multi-task dense prediction, parameter-efficient fine-tuning, low-rank adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Nx6XOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ISC-LoRA, a training-time conditioning scheme for multi-task LoRA with shared and task-private branches. Inside each task-private branch, the bottleneck features P = AX are modulated by a fixed radial spectral tilt T. The modulation strength is σ·κ(P, A), where κ ∈ [−1, 1] is a stop-gradient scalar computed for each sample from the covariance and the frequency-weighted covariance of P. The ridge regularization uses AAᵀ, which makes κ invariant to changes of the LoRA rank basis. A cosine continuation schedule then reduces σ (σ_max = 0.1) to zero during training, so the deployed model is an ordinary LoRA with no extra parameters and no inference cost. The paper proves forward rank-basis invariance and a bound on the perturbation. It evaluates on PASCAL-Context, NYUD-v2, and ten Cell Tracking Challenge datasets, using Swin-T and DINOv3 backbones (ViT-L/16 and ViT-7B/16).

Critical strengths

Critical Strengths

  1. Simple, deployment-free design. The spectral response exists only during training and is removed by the end of it, so the final model is a standard LoRA with no extra parameters and no inference overhead. This makes the method easy to adopt if its benefit holds up.
  2. Gains over ordinary LoRA in matched settings. In Table 3, ISC-LoRA improves all seven metrics over ordinary LoRA in two settings (Swin-T and DINOv3-7B), with the same backbone, heads, and training budget.
Critical weaknesses
  1. The evidence does not support the claimed mechanism, and the gains are small.
    • Reversed ISC, which flips the sign of κ, retains most of the improvement (Swin-T: +0.47 vs. +0.64 mIoU; DINOv3-7B: +0.14 vs. +0.18).
    • κ is almost always negative (Fig. 3a), so the claimed input- and task-dependent spectral selection is not what drives the results.
    • Gains over ordinary LoRA are marginal (+0.18 to +0.64 mIoU; 0.001 to 0.007 depth RMSE), come from single runs without variance, and lack simple controls such as a fixed κ, a random-sign κ, or feature noise.
    • In the DINOv3 block, ordinary LoRA alone already outperforms the reproduced CvM and M2H-MX, so the advantage shown in Table 2 comes mainly from the training recipe.
  2. Poor readability. The writing is dense and full of obscure, self-coined terminology ("intrinsic," "continuation," "subspace-conditioned," "rank-basis orbit"). The text reads as insufficiently refined (the authors disclose LLM-assisted writing). Simple ideas are wrapped in heavy formalism, which makes the actual mechanism hard to follow.
  3. Missing discussion of and comparison with related work.
    • The multi-task dense prediction architecture literature is not discussed (e.g., PAD-Net, MTI-Net, ATRC, InvPT, TaskPrompter, BridgeNet, TaskExpert, MLoRE), although the motivation of task- and input-dependent spatial adaptation is closely related.
    • It is also unclear whether ISC could be applied to, or extend, such architecture-based methods.
    • Standard reference points for PEFT multi-task papers are missing: trainable parameter counts, single-task and multi-task full fine-tuning baselines, and Δm.
  4. CTC is not strictly dense prediction and does not test the multi-task setting. Only the flow and probability maps are dense outputs. DET and TRA are derived from them through fixed post-processing. Using one private branch per dataset with a shared head is multi-domain adaptation, not multi-task learning, so the CTC results add little support to the paper's central claim.
Questions for authors
  1. Reversed ISC keeps most of the gain, and κ is almost always negative (Fig. 3a). Would a constant κ or a random-sign κ match ISC? If so, what role do the subspace-derived direction and the input dependence actually play?

  2. The response is fully removed before evaluation. What does ISC change in the learned A_t and B_t compared with ordinary LoRA? Please also report the mean and standard deviation over several seeds for Table 3.

  3. In Table 2, ordinary LoRA alone already outperforms the reproduced CvM and M2H-MX. Were these baselines trained with the same recipe? Please add ordinary LoRA as a row.

  4. Can ISC be applied to architecture-based multi-task methods (e.g., BridgeNet or TaskPrompter decoders) or under full fine-tuning? Please also report trainable parameter counts and Δm.

  5. In CTC, all private branches share the same output, so what does "task-private spectral context" mean in this setting? Do the κ values differ across datasets?

Reviewer_dK7xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ISC-LoRA. The idea is to apply input-dependent spectral conditioning to low-rank adapters only during training, and gradually fade it out so inference keeps the standard zero-cost LoRA format. The authors show rank-basis invariance theoretically and test on dense prediction and cell tracking benchmarks.

Critical strengths

The training-only spectral regularizer is neat since it avoids inference overhead completely.

Solid math grounding on rank-basis invariance and stability bounds.

Tested across multiple vision backbones and datasets, including DINOv3 and cell tracking.

Critical weaknesses

The gains look very marginal. In Table 1, lots of improvements are just 0.01 or 0.02. Table 3 is similarly borderline over regular LoRA, especially on DINOv3. With no standard deviations or multiple seeds provided, it is hard to tell if this is real signal or just random seed noise.

Not clear why this specific subspace-aware design is necessary. The ablation doesn't show if the gains come from your exact frequency selection or just generic spectral noise injection / perturbation during training.

No training wall-clock or memory numbers. Even if inference cost is zero, if training slows down a lot, that matters.

Questions for authors

Can you provide error bars or mean/std over 3-5 random seeds for Tables 1 and 3?

What happens if you replace your adaptive frequency selection with a simple fixed bandpass or random spectral perturbation under the same annealing schedule?

What is the actual training overhead (wall-clock time and peak VRAM) compared to vanilla LoRA?

Reviewer_r6sJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ISC-LoRA, a training-time spectral conditioning method for multitask dense prediction. It derives an input- and task-dependent scalar from each private LoRA branch using subspace geometry and covariance-normalized frequency statistics. This scalar controls a bounded radial spectral response, whose strength is gradually reduced to zero, allowing standard LoRA inference. The paper proves forward rank-basis invariance and bounds the local output perturbation. Experiments cover PASCAL-Context, NYUD-v2, and ten CTC datasets with Swin-T and DINOv3 backbones.

Critical strengths

1 The paper identifies a concrete issue in conditioning LoRA features: equivalent rank-basis representations can produce different energy-based spectral responses. Using the geometry of the private subspace addresses this ambiguity directly.

2 The mathematical properties are clearly stated and supported by proofs. Theorem 3.1 establishes forward rank-basis invariance, while Proposition 3.2 bounds the spectral gain and private-output perturbation. The paper also explicitly distinguishes these guarantees from invariance of optimization trajectories.

3 This paper is easy to follow and understand.

Critical weaknesses

1 The benefit of input- and task-specific response selection is not fully isolated. The method reduces the measured context to one scalar controlling a fixed radial response. Figure 3 demonstrates coefficient variation, but does not establish that this variation improves prediction. Fixed-coefficient or shuffled-context controls would help distinguish useful contextual adaptation from generic spectral regularization.

2 Several reported advantages are small, without uncertainty estimates. In Table 1, ISC-LoRA exceeds Free Sinewich by only 0.01 percentage points on both part segmentation and saliency. In Table 3, Swin-T depth RMSE changes from 0.617 to 0.616. The paper does not report the average relative multi-task performance.

3 Two central training choices remain insufficiently validated. The paper defines stopping gradients through the statistic and gradually withdrawing the response as parts of ISC-LoRA, but Table 3 does not separately test either choice. The existing conditioner comparisons therefore do not establish whether detachment and continuation are necessary for the reported gains.

Questions for authors

1 Does ISC-LoRA outperform a fixed spectral coefficient or a coefficient shuffled across inputs or tasks? Separately, what happens when gradients pass through the statistic, or when gradual withdrawal is replaced by constant-strength conditioning followed by removal?

2 For each configuration, when does withdrawal begin and end, and is there any identity-only training afterward? What exact validation aggregate selects checkpoints, and were the selected checkpoints obtained after withdrawal was complete?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.