acceptodds
Under review as a conference paper at ICLR 2027

APERTURE: What to measure, how much to measure, and where to allocate in multimodal low-rank adaptation

Abstract

The same multimodal input may require different evidence depending on the question, yet conventional low-rank adaptation (LoRA) exposes the same fixed set of measurements regardless of the query. Under a limited adaptation budget, this fixed allocation can miss question-relevant evidence, provide insufficient within-modality support, and prevent capacity from shifting across modalities as their relative importance changes. We formulate this challenge as question-conditioned measurement allocation: jointly deciding what to measure, how much to measure, and where to allocate a shared average measurement budget. We introduce Aperture, a framework that operates on a fixed bank of reference and gradient-informed candidate directions. Question-conditioned rotations adapt measurement directions, gates control the amount of exposed support within each modality, and a joint support regularizer redistributes the shared budget across modalities. In experiments with LLaMA-2-7B, Aperture outperforms the question-blind Static-Alloc control by 4.18 percentage points on the official MUSIC-AVQA split and 2.23 points in AVE segment accuracy, at matched mean support and trainable parameter count. On a video-disjoint MUSIC-AVQA split, it retains a 2.09-point gain. These results demonstrate the benefit of making measurement allocation question-dependent under a fixed adaptation budget. Code is available at https://anonymous.4open.science/status/Aperture-F0C0.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, Question-Conditioned Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jAwfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a novel framework for low-rank adaptation that operates on a fixed bank of reference and gradient-informed candidate directions. The idea sounds and the results are interesting.

Critical strengths

the idea is very interesting.

according to the results, aperture seems very effective and useful for the community.

Critical weaknesses

The paper lacks discussions and comparisons with two important related works, namely CCM-LoRA and Gated LoRA. A comparison with these methods would consistently strengthen the contribution of the paper.

Notation should be revised: vectors and matrices should be bold, otherwise they represent scalars!

Sec 5.1 could be revised to be less-llm generated. Also, I got a little bit lost among RQ1,2,3,4, tier 1,2,3, and stage 1,2.

While the paper seems nicely motivated and presented, the experimental section seems written in rush. So, even though the method seems effective, the paper lacks a detailed discussion on the results. Also there is space left, so it would have been nice to place some visualizations to qualitatively evaluate the impact of aperture. I think this part could be improved to improve the quality of the paper.

Minor concerns:

Formatting should be revised. figure 1 is placed at the bottom overlapping with page number.

controlled empirical validation is not a contribution.

The policy implemented by Aperture in section 3 should be moved to the method section.

Questions for authors

could the authors compare with the two relevant literature methods in the experiments and discuss the results?

4 comments
AuthorsOpenReview ↗

Response to Reviewer jAwf

We have carefully addressed all concerns through additional comparisons with CCM-LoRA and Gated LoRA and allocation-swap experiments, clarification of tensor notation and the distinction between the problem formulation and the method, and improvements to the experimental organization, results discussion, qualitative examples, figure layout, and contribution statement. The corresponding revisions have been incorporated into the manuscript.

Paired results are MUSIC-AVQA/AVE; gains are percentage points (pp).

W1 & Q1 (CCM-LoRA and Gated LoRA).

CCM-LoRA routes inputs among existing rank directions under an expected-rank budget; Gated LoRA gates rank-one mini-experts. APERTURE additionally rotates directions toward gradient-informed residuals and allocates shared support across audio and vision.

We compared both methods on LLaMA-2-7B (seed 42), matching the split, update budget, and validation-based selection while retaining their native architectures and capacity settings:

MethodMUSIC-AVQAAVE
CCM-LoRA [1]73.4871.15
Gated LoRA [2]73.7271.34
Static-Alloc$^+$73.3773.35
APERTURE77.5575.58

Gains are +4.07/+4.43 pp over CCM-LoRA (video-clustered 95% CIs [3.21,4.93]/[3.29,5.57][3.21,4.93]/[3.29,5.57]) and +3.83/+4.24 pp over Gated LoRA ([2.97,4.69]/[3.10,5.38][2.97,4.69]/[3.10,5.38]). Static-Alloc$^+$ and APERTURE each train 3,507,712 parameters; APERTURE averages eight active rows per site.

Matched-support controls (±0.10 rows per modality) show that random residual directions, zero rotation, and separate modality budgets reduce accuracy by 4.31/4.40, 2.70/3.43, and 1.43/1.20 pp. Some main-table bank/zero-angle rows also differ in budget control, so they do not isolate independent causal effects.

Revision. Sec. 2 (p. 3), Sec. 5.1 (p. 8), Sec. 5.2 (p. 9); Tables 7 (p. 28), 21–23 (pp. 39–40).

[1] Rafiuddin and Abdullah, Context-Conditioned Masked LoRA, Findings of ACL 2026.

[2] Eom et al., Gated LoRA: Dual-Purpose Projections for Parameter-Efficient Mini-Expert Fine-Tuning, ICLR 2026 submission.


W2 (Notation).

We standardized bold uppercase matrices, bold lowercase vectors, and italic scalars, with dimensions at first use. Eq. 9 now clarifies that unit directions are scaled by reference row norms before learned gating.

Revision. Secs. 3–4.4 (pp. 4–7); Apps. A.1 (p. 16), A.4/A.6.

W3 (Organization).

We replaced RQ labels with an experimental overview, Tier labels with backbone names, and S1–S3 with direction orientation, within-modality support, and cross-modal allocation. Stage 1/2 now denote reference-adapter/allocator training only. Sec. 5.1 explains each control's purpose; Figure 2's labels and matrix notation match the text.

Revision. Sec. 5.1 (p. 7), Secs. 5.2–5.4 (pp. 8–10), App. B (pp. 24–35), Figures 1–2 (pp. 2–5).


W4 (Interpretation and qualitative evidence).

We swapped allocations between audio/visual questions about the same video while fixing weights, rotated directions, row rankings, and the eight-row budget. The 1,420 unlabeled pairs comprise 2,840 evaluations across 612 videos (64.3% of eligible pairs), selected for allocations differing by at least two audio rows.

Swapping reduces accuracy 76.83→71.41%, a 5.42 pp drop (video-clustered bootstrap 95% CI [3.86,6.98][3.86,6.98]), with 214 right-to-wrong versus 60 wrong-to-right flips.

For video 00005455, the audio question “Which instrument makes sounds before the violin?” uses 6 audio + 2 visual rows and correctly predicts piano; swapping to 2+6 changes the answer to violin. The visual question “What is the instrument on the left of the violinist?” uses 2+6, correctly predicts piano, and changes to cello under 6+2. Thus the same video's correct answers depend on question-appropriate allocation.

These interventions complement question-shuffling controls and support–demand correlation (ρ=0.46\rho=0.46, versus 0.08 with shuffled questions; Table 2b).

Revision. Sec. 5.2 (p. 9), Sec. 5.4 (pp. 9–10); Tables 28–29 (App. C.6, pp. 42–43).


Minor 1 (Figure layout). Figure 1 is now at the top of p. 2, clear of the footer.

Minor 2 (Contribution). The third contribution now states the three-seed mean gains +4.16 pp on MUSIC-AVQA and +2.23 pp on AVE over parameter-matched Static-Alloc$^+$ (Sec. 1, p. 3).

Minor 3 (Formulation versus method). Sec. 3 (p. 4) gives the general measurement/allocation formulation; method-specific rotations, gates, and budget regularization appear in Secs. 4.2–4.3 (pp. 5–6).

We hope these analyses address your concerns.

Maybe you forgot to upload the updated the paper? I can't see the changes.

AuthorsOpenReview ↗

reply to reviewer jAwf

I have now uploaded the updated PDF. Thank you for pointing this out, and I apologize for the oversight. I hope our responses and revisions have adequately addressed all your concerns. If you have any further questions or suggestions, we would be happy to clarify them.

Thanks for the rebuttal. I appreciate the new experiments. Nevertheless, the review is still totally llm-based, making the paper not ready for a conference as the writing quality does not match the required quality for publishing at ICLR. I show here some examples to help the authors understand that submitting and publishing papers should be taken seriously and that it is important to study how papers should be done before asking llms to write/do/revise them. Just a couple of examples:

  • caption of fig.1, "The short labels inside the allocator panel now name the three decisions." Now? it was supposed to be a ready paper, not a report on the changes

  • Third contribution starts with "across three seeds". This cannot be a contribution, this is a mere implementative detail that is standard, while llms tend to be over-specific on implementation details, this is a scientific paper and experimental details should only go in the experimental section.

  • The experimental discussion lacks, as llms just report some results like "APERTUREgains+2.03/+1.24, +0.72/+1.28,and+3.06/+1.99points",which is a useless information as it is already contained in the tables, while instead one expect a discussion or a comment on the results, rather than just replicating the numbers.

Overall, I find that the paper is not ready for this conference.

Reviewer_ZABdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

APERTURE makes multimodal LoRA question-dependent. It rotates candidate measurement directions and gates how many are active in each modality under a shared average budget. Its fixed bank combines reference-LoRA directions with complementary directions selected using training-only loss gradients. Experiments report promising results across different multimodal settings.

Critical strengths
  1. The paper identifies a meaningful problem: different questions require different evidence, whereas conventional LoRA exposes a fixed set of measurement directions. Framing this as deciding which directions to use, how much support to activate, and how to allocate a shared budget offers a reasonably clear approach to question-conditioned multimodal adaptation.

  2. The method separates these decisions into interpretable components: a fixed candidate bank, question-conditioned rotations, gates, and a shared average-support constraint. Reference-preserving initialization and component ablations make the design easier to inspect and provide useful evidence about the contribution of its main components.

Critical weaknesses
  1. The motivation remains largely intuitive. Although APERTURE improves over Static-Alloc+, the paper does not directly show when fixed, question-independent LoRA measurements miss relevant evidence or establish that this is a substantial bottleneck.
  2. The experimental validation of the method is not comprehensive enough. Comparisons with external methods are limited to LLaMA-2-7B. Results on other backbones mainly compare APERTURE with Static-Alloc+. In addition, key parameter-count and seed-level records remain unaudited.
Questions for authors
  1. Could the authors provide a targeted analysis of when the question-blind reference LoRA misses question-relevant evidence, and whether APERTURE’s question-conditioned allocation corrects those same cases under matched support?
  2. Does APERTURE retain its advantage over external baselines beyond LLaMA-2-7B? Evaluating APERTURE against the baselines on additional backbones would provide more convincing evidence of its effectiveness.
  3. Appendix Table 12 (p. 29) flags the parameter-share figures for audit, while Appendix Table 16 (p. 33) marks several supporting records or audits as pending. Could the authors provide a clearer explanation of the status and implications of these items?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ZABd

We have carefully addressed all concerns through additional experiments on question-relevant evidence mismatch, allocation interventions, and external baselines across additional backbones, clarification and validation of parameter counts, seed-level statistics, and evaluation protocols, and improvements to the empirical motivation and reproducibility documentation. The corresponding revisions have been incorporated into the manuscript.

Paired results below are MUSIC-AVQA/AVE; gains are percentage points (pp).

W1 & Q1 (Evidence mismatch).

On the disjoint audit set C2\mathcal C_2 (1,024 training-derived examples excluded from bank construction), APERTURE captures 0.21/0.27 audio/visual gradient-energy headroom, including 0.08/0.12 from gating beyond rotation. Mean support is matched near eight rows: Static-Alloc$^+$ 4.02+3.99, APERTURE 4.10+3.90.

Stratification uses reference-host demand mismatch ∣sv−0.5∣|s_v-0.5|, independently of learned gates. Fixed/Broken denotes errors corrected/correct answers flipped:

MismatchnFixed/Broken vs. Static-Alloc$^+$Gain (pp)
<0.1043723/14+2.06
0.10–0.2535928/13+4.18
>0.2522826/9+7.46
All1,02477/36+4.00

Against the reference, APERTURE corrects 106/322 errors and flips 36/702 correct answers, gaining +6.84 pp. High-minus-low stratum gains over Static-Alloc$^+$ are +5.40 pp (95% CI [−0.29,11.08][-0.29,11.08]), or +4.27 pp after template standardization ([−1.55,10.09][-1.55,10.09]). Both intervals include zero; the stratum trend is not conclusive.


W1 & Q1 (Allocation intervention).

We froze weights, rotated directions, and row rankings, fixed eight binary active rows per site, and changed only the modality split:

SplitMUSIC-AVQAAVEFixed/Broken vs. reference
Learned74.8075.49101/37
Equal 4+473.1474.1289/42
Permuted71.1972.5876/49

Learned allocation gains +1.66/+1.37 pp over equal allocation and +3.61/+2.91 pp over permutation (video-clustered 95% CIs [1.26,5.97]/[0.79,5.03][1.26,5.97]/[0.79,5.03]). Hard-mask discretization explains 101/37 versus 106/36 and micro accuracy 74.80% versus 75.39%; Table 2's 75.16% is a question-group macro average.

Revision. Sec. 5.4 (pp. 9–10); Tables 24–25 (App. C.3, p. 41).

W2(a) & Q2 (External baselines on additional backbones).

At seed 42, data, Stage-1 checkpoints, Stage-2 updates, and validation-based selection are matched; baselines retain their native architectures and Table 7 capacity settings. Full results are in Table 26:

BackboneAPERTURE accuracyGain over strongest external baseline
Qwen3-8B83.82/79.38+2.03/+1.24
LLaMA-3-8B79.63/77.66+0.72/+1.28
Qwen2-7B76.64/73.62+3.06/+1.99

Video-clustered 95% CIs for these gains are [1.17,2.89]/[0.10,2.38][1.17,2.89]/[0.10,2.38], [−0.14,1.58]/[0.14,2.42][-0.14,1.58]/[0.14,2.42], and [2.20,3.92]/[0.85,3.13][2.20,3.92]/[0.85,3.13], respectively. Five of six exclude zero. Including LLaMA-2-7B (+3.75/+4.22 pp), APERTURE leads all eight host–dataset pairs in observed accuracy. Reference-cont and Static-Alloc$^+$ are internal controls, excluded from the external-baseline comparison.

Revision. Sec. 5.2 (p. 9); Table 26 (App. C.4, p. 42).


W2(b) & Q3 (Completed audits).

Parameters: analytical counts (Eq. 32 plus biases) match checkpoints. Static-Alloc$^+$ and APERTURE each train 3,507,712 parameters (approximately 0.05% of 7B); only the conditioner input differs. Static-Alloc trains 3,245,632. Table 12 replaces the draft 1.08%/1.12% estimates.

Seeds: MUSIC-AVQA scores for seeds 42/43/44 are 73.37/73.36/73.35 for Static-Alloc$^+$ and 77.55/77.40/77.62 for APERTURE. Mean ± sample SD gains are +4.16±0.12+4.16\pm0.12 pp (MUSIC-AVQA) and +2.23±0.13+2.23\pm0.13 pp (AVE). The seed-averaged MUSIC-AVQA gain has video-clustered 95% CI [3.30,5.03][3.30,5.03]. AVE per-seed absolute accuracies are not tabulated.

Protocols: Table 16's evaluation, calibration, control, and cost records are complete. Video-disjoint accuracy rises 74.88→76.97 (+2.09 pp), using 6,000/1,431/1,857 train/validation/test videos. Construction/audit sets are disjoint, each with 1,024 examples; support is calibrated within ±0.10 per modality. Static-Alloc$^+$/APERTURE latency is 1.04×/1.07× and memory 1.03×/1.05× LoRA. Code and artifacts will be released upon publication.

Revision. Tables 12 (p. 31), 16 (p. 35), and 27 (App. C.5, p. 42); Apps. B.2/B.10.

We hope these analyses address your concerns.

Thank you for the detailed response and additional analyses. The rebuttal addresses some of my concerns. However, I still have concerns about the rigor of the experimental reporting. Overall, I maintain my score.

Reviewer_j87AOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces APERTURE, a framework for question-conditioned measurement allocation in multimodal low-rank adaptation. Using a fixed bank of reference and gradient-informed directions, it combines conditional rotations, gating, and shared-budget regularization to adjust measurement directions and allocate capacity across modalities. Experimental results show improved performance over the question-blind baseline.

Critical strengths
  1. Clear presentation and motivation. The paper clearly explains the limitations of fixed measurement allocation and organizes its approach around what to measure, how much to measure, and where to allocate capacity. This makes the motivation and method easy to follow.

  2. Promising experimental results. APERTURE improves over the question-blind baseline across several tasks and backbones, including a video-disjoint evaluation. The static, full-bank, and question-shuffling controls also provide useful evidence for the benefit of question-conditioned allocation.

Critical weaknesses
  1. The parameter-matching claim is not fully verified. The paper states that Static-Alloc⁺ and APERTURE have identical trainable tensors, but Table 12 reports different trainable parameter shares: 1.08% and 1.12%, respectively. The appendix acknowledges this inconsistency and marks the values as pending verification. Since parameter matching is central to attributing the improvement to question-conditioned allocation rather than additional capacity, the paper should resolve this discrepancy and provide verified parameter counts.

  2. The fixed candidate bank may limit adaptation as the model changes during training. The complementary directions are selected using gradients from the Stage-1 reference model and remain frozen throughout Stage 2, even though the output mappings and other adaptation components continue to change. The paper does not establish whether directions selected for the initial reference model remain useful throughout this process. Further justification or sensitivity analysis would help assess the consequences of fixing the bank, including its dependence on the calibration examples used for construction.

Questions for authors

The paper interprets LoRA input projections as measurements, while Appendices A.6 and A.13 acknowledge that conditional mixing over the complete candidate bank can reproduce the same linear projection. Beyond this interpretation, could the authors clarify the specific methodological advantage of the proposed rotation-and-gating parameterization over existing conditional mixing or subspace-selection approaches? In particular, does its benefit arise from a useful inductive bias, regularization, or easier optimization, and what evidence distinguishes these explanations?

1 comment
AuthorsOpenReview ↗

Response to Reviewer j87A

We have carefully addressed all concerns through additional experiments on fixed-bank stability, calibration sensitivity, and conditional mixing, clarification and validation of exact trainable-parameter matching, and improvements to the explanation of the rotation-and-gating parameterization and its optimization, inductive-bias, and regularization effects. The corresponding revisions have been incorporated into the manuscript.

W1 (Parameter matching).

The draft 1.08%/1.12% shares were preliminary estimates. Module-shape calculations (Eq. 32 plus biases) and checkpoint audits independently verify that Static-Alloc$^+$ and APERTURE each train exactly 3,507,712 Stage-2 parameters, including inherited text projections and output maps. Their tensors are identical; only the conditioner input differs: the training-mean versus current-question embedding. Static-Alloc trains 3,245,632 parameters.

At matched capacity, APERTURE gains +4.18/+2.23 pp on seed 42 and +4.16±0.12+4.16\pm0.12/+2.23±0.13+2.23\pm0.13 pp across seeds 42–44 (MUSIC-AVQA/AVE; mean ± sample SD). With identical tensors, shuffle-train scores 72.16/70.08 versus 73.37/73.35 for Static-Alloc$^+$; test-time question shuffling reduces audit accuracy from 75.16 to 70.66.

Revision. Table 12 (p. 31); Apps. A.9 (p. 19), A.16, and B.10 (pp. 33–35).


W2 (Fixed bank and calibration sensitivity).

The bank fixes Stage-1 basis directions, while Stage 2 learns question-conditioned rotations and gates. Construction and audit sets are disjoint. Candidates use X⊤Δ\mathbf X^\top\boldsymbol\Delta without B0\mathbf B_0, which Stage 2 updates; including that map lowers accuracy from 77.55/75.58 to 76.20/74.45 (Table 10).

Frozen-bank gradient coverage declines from 0.83/0.80 to 0.78/0.75 across training (audio/visual), versus 0.82/0.79 under full reselection. Endpoint rebuilding and retraining yields 77.64/75.65, only +0.09/+0.07 pp, below bank-draw SDs of 0.13/0.14 pp. This does not establish statistical equivalence, and the endpoint arm requires an additional source training run. Epoch-wise refresh scores 77.41/75.44 versus 77.33/75.38 under matched resets, a +0.08/+0.06 pp difference.

For calibration sizes 256/512/1,024/2,048, mean accuracies are 77.14/77.39/77.55/77.61 on MUSIC-AVQA and 75.12/75.38/75.58/76.65 on AVE. Increasing 1,024 to 2,048 examples gains +0.06 pp on MUSIC-AVQA, within its 0.13 pp draw SD, but +1.07 pp on AVE, above its 0.14 pp draw SD. Thus calibration sensitivity is task-dependent. Mean principal-angle cosine increases from 0.79 to 0.95.

Revision. Sec. 4.1 (p. 5); Tables 18–20 (pp. 37–38).


Q1 (Advantage over conditional mixing).

A full-bank conditional mixer is theoretically at least as expressive (Apps. A.6/A.13). The benefit is therefore the parameterization, rather than a larger function class. At seed 42, APERTURE uses 8/16 active rows and outperforms a mixer with matched downstream width by +2.73/+1.70 pp (77.55/75.58 versus 74.82/73.88). Controls distinguish the following effects, which are not additive:

  • Optimization: reference-preserving initialization gains +1.52/+1.88 pp, lowers validation loss from 0.418/0.435 to 0.371/0.385, and reaches the target loss in 36% fewer updates.
  • Inductive bias: structured parent–residual pairing gains +1.04/+0.72 pp over the comparison configuration and +0.82/+0.61 pp over random pairing at equal coefficient count. Bounded rotation gains +0.87/+0.66 pp over a free paired mixer; parameterization and angle-bound effects overlap.
  • Regularization/allocation: tightening bounds from unbounded to π/4\pi/4 raises accuracy from 76.78/74.90 to 77.55/75.58 and reduces train–validation gaps from 4.28/4.15 to 3.12/3.02 pp. Removing the budget penalty expands support from 8.01 to 12.38 rows and lowers both accuracies by 1.45 pp. Shared versus separate budgets gain +1.43/+1.20 pp at equal total support; top-gate versus random rows gain +2.92/+2.84 pp.

Revision. Sec. 4.2 (p. 6); Tables 21–22 (pp. 39–40); Table 10 (p. 30).

We hope these analyses address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.