acceptodds
Under review as a conference paper at ICLR 2027

Geofield-LoRA: Reading Task-Specific Adapters from a Shared Gaussian Primitive Field

17 views

Abstract

Low-Rank Adaptation (LoRA) provides an efficient way to adapt large language models, but multi-task adaptation must capture both the commonalities and differences across task-specific updates. Existing conditional LoRA methods typically address this through learned mixing, routing, or parameter-generation functions. In practice, such functions can concentrate most tasks on a few shared components. We instead formulate multi-task adaptation from a multi-view perspective, where different tasks obtain related but distinct updates from a shared parameter field. We introduce Geofield-LoRA, which represents shared low-rank updates as Gaussian primitives in a high-dimensional task-view space. A task condition defines a view of this shared field, and ray-based Gaussian responses determine how the primitive-specific low-rank updates are combined into a task-specific LoRA residual. This allows different tasks to reuse the same low-rank component bank while forming distinct task-specific updates through different combinations. Extensive and controlled studies show that Geofield-LoRA makes effective use of shared low-rank capacity: tasks with related adaptation requirements share primitives while retaining distinct updates, highlighting view-conditioned parameter fields as a promising direction for multi-task adaptation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Task Learning, Low-Rank Adaptation (LoRA), Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JzHhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Geofield-LoRA introduces a multi-view formulation for multi-task Low-Rank Adaptation (LoRA) in large language models. Rather than treating individual tasks as independent requests for adapter parameters or relying on implicit routing and mixing functions that can cause tasks to concentrate on a small number of shared components, Geofield-LoRA treats tasks as different perspectives on a shared adaptation space.

Critical strengths

Multi-View Formulation: Re-conceptualizes multi-task adaptation by mapping task conditions (composed of descriptions, answer schemas, and support examples) into a high-dimensional task-view space, treating task commonalities and differences as views of a shared parameter field.

Gaussian Low-Rank Primitive Field: Represents shared low-rank updates using Gaussian primitives (defined by centers, precision matrices, and strengths) within each layer. Task views project inward rays into this space, and the normalized geometric responses determine how primitive-specific low-rank factors are combined into task-specific LoRA residuals.

Extensive Empirical Validation: Evaluated across multiple backbones (Qwen2.5-1.5B, Gemma-2-2B, and Llama-3.2-3B) on the CLEAN56 benchmark subset, outperforming shared LoRA, dense MLP mixing, sparse top-kk routing (MixLoRA), and other multi-task baselines while maintaining higher task diversity and lower training-to-test gaps.

Critical weaknesses

Increased Training Overhead: While inference efficiency matches standard conditional routing (since task views are cached per task), the training time for Geofield-LoRA is noticeably higher (e.g., 4.73 hours compared to 3.26–3.51 hours for baselines) due to the optimization of Gaussian geometry and view-conditioned responses.

Generalization to Unseen Tasks Discussed Only Hypothesically: Although the discussion touches on leveraging text-to-LoRA or diffusion generators to produce views for unseen tasks, the paper lacks empirical evaluation on zero-shot task adaptation using the learned field.

Questions for authors

Unseen Task Generalization: How sensitive is the trained primitive field to out-of-distribution or entirely unseen task views during inference? Can a newly introduced task condition be mapped into the existing task-view space via WviewW_{view} without retraining the Gaussian primitives or low-rank factors?

Computational Scaling: As the number of tasks TT scales to hundreds or thousands (beyond the 56 tasks in CLEAN56), does the optimization of the high-dimensional task-view space and primitive geometry exhibit stability issues or bottlenecks?

Reviewer_wZyxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Geofield-LoRA for multi-task adaptation. It represents shared low-rank updates as Gaussian primitives in a high-dimensional task-view space. Task conditions define different views, and ray-based Gaussian responses determine task-specific LoRA updates. Experiments across three backbones and matched low-rank budgets show improved test scores and more distinct task-specific updates.

Critical strengths
  1. The multi-view formulation provides an interesting perspective on shared and task-specific adaptation.
  2. The paper provides controlled comparisons under matched low-rank budgets and analyzes task–primitive combination weights and task-specific LoRA updates.
Critical weaknesses
  1. Limited score gains. In Table 1(a), Geofield-LoRA shows only small improvements over MixLoRA on Qwen2.5-1.5B in Test (+1.09), Train (+0.18), and Gap (−0.91). The 95% confidence interval for the Test difference includes zero (Table 1(b)). The advantage of the proposed geometric formulation remains insufficiently supported.

  2. Limited justification of the geometric formulation. The method combines shared low-rank components through task-conditioned weights, similar to existing conditional LoRA methods. Although the component ablations show improvements (Table 5), the necessity of Gaussian primitives and ray-based responses over learned mixing or routing remains unclear.

  3. Limited generalization. The experiments only cover a single benchmark (i.e., CLEAN56) and three relatively small backbones (1.5B–3B). The test templates belong to the same tasks as the training templates. Generalization to other benchmarks, larger backbones, and unseen tasks remains unclear.

Questions for authors
  1. How do the authors explain the small performance gains over MixLoRA on Qwen2.5-1.5B, especially given that the 95% confidence interval includes zero?

  2. What is the key advantage of Gaussian primitives and ray-based responses over existing learned mixing or routing methods?

  3. Can the proposed task-view space support adaptation to unseen tasks and other benchmarks?

Reviewer_8Fy4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Geofield-LoRA for multi-task adaptation. Each layer contains Gaussian primitives in a learned task-view space; a task condition, built from a task description, answer schema, and support examples, is encoded into a view. Ray-based Gaussian responses determine how primitive-specific low-rank updates are combined into a task-specific LoRA update.

Critical strengths
  1. The paper introduces a geometric perspective on parameter sharing, treating task-specific adapters as different views of a shared Gaussian primitive field. This provides an interesting alternative to conventional routing-based and mixing-based approaches.
  2. Geofield-LoRA achieves the best performance across Qwen2.5-1.5B, Gemma-2-2B, and Llama-3.2-3B, outperforming several representative multi-task LoRA baselines.
  3. The paper goes beyond reporting task performance by analyzing primitive utilization, task-specific update similarity, and effective dimensionality. These analyses provide useful evidence that Geofield-LoRA produces more diverse task-specific adaptations.
Critical weaknesses
  1. The motivation for Geofield-LoRA is not sufficiently convincing. The paper argues that conventional routing methods tend to concentrate tasks on a few shared components and proposes a geometric routing mechanism to encourage more diverse expert combinations. However, expert concentration is not necessarily a problem in multi-task learning, as tasks with similar adaptation requirements may naturally benefit from sharing components. Moreover, concentrated expert usage could facilitate LoRA pruning or expert merging, potentially improving parameter efficiency. Therefore, the paper does not sufficiently justify why expert concentration should be treated as a limitation that needs to be addressed, making the connection between the identified problem and the proposed method less compelling.
  2. All three evaluated backbones in the main cross-backbone study are 1.5B, 2B, or 3B; there is no 7B evaluation. Given the paper's LLM focus, this leaves the method's effectiveness and overhead at the commonly used 7B scale unverified, and limits confidence in broader large-model applicability.
  3. The use of unusually large LoRA ranks raises concerns about the baseline comparisons. In the main experiments, Shared LoRA uses a rank of 512 to match the total low-rank budget of 32 rank-16 experts. However, conventional LoRA typically uses much smaller ranks, such as 16, or 32, which can often achieve competitive or even better performance. Moreover, Table 2 shows that increasing the Shared LoRA rank from 64 to 512 provides almost no improvement. This raises concerns about whether the selected baseline configurations adequately reflect the performance of properly tuned LoRA. I encourage the authors to include comparisons with smaller, commonly used LoRA ranks to provide a more comprehensive evaluation.
  4. Some important training hyperparameters appear to be missing. Although the paper specifies the batch size and number of training epochs, it does not clearly report the learning rate, warmup configuration, or learning rate scheduler.
Questions for authors
  1. The paper treats greater primitive diversity and more balanced component utilization as evidence of more effective multi-task adaptation. However, concentrating multiple tasks on a small number of shared components is not necessarily undesirable, particularly when these tasks have similar adaptation requirements. Could the authors provide evidence that the observed concentration in conventional routing methods actually leads to negative transfer or insufficient adaptation capacity? Furthermore, is there a direct relationship between increased primitive diversity and improved downstream performance?
  2. How well does Geofield-LoRA scale to larger LLMs, such as 7B models, in terms of performance and computational overhead?
  3. Could smaller, commonly used LoRA ranks, such as 16 or 32, achieve comparable or better performance than the rank-512 Shared LoRA baseline? Can the authors include these comparisons?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.