acceptodds
Under review as a conference paper at ICLR 2027

TALE: Training-Aware Low-Rank Emulation for On-Device Adaptation

Abstract

Fine-tuning foundation models on edge devices remains challenging because training often requires 3–4 more memory than inference. Emulator-based approaches address this challenge by using a lightweight surrogate for the full model, enabling adaptation within an inference-level memory budget. Existing emulators are typically constructed to preserve the full model’s forward outputs. However, forward fidelity alone does not ensure that the resulting emulator reproduces the training signals needed for fine-tuning. To address this, we propose TALE, a Training-Aware Low-rank Emulation for memory-efficient on-device adaptation. Specifically, our theoretical analysis reveals that adapter divergence is fundamentally governed by the gradient discrepancy between the emulator and the full model. Guided by this insight, TALE optimizes a direct gradient-matching objective, extracting the compact weights in closed form via SVD. This principled approach intrinsically aligns with the full model's loss geometry, yielding an emulator that faithfully replicates the training signals essential for adaptation. Across multimodal VQA, LLM personalization, and image generation, TALE consistently outperforms forward-only emulators and achieves performance competitive with full-model fine-tuning. Furthermore, it demonstrates robust generalization capabilities to unseen users and tasks, supporting decoupled server-side generation for on-device use.

Primary area:
foundation or frontier models, including LLMs
Keywords:
parameter-efficient fine-tuning, foundation model adaptation, low-rank approximation, on-device training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9cwYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose TALE, a framework for memory-efficient model adaptation on edge devices that constructs a compressed emulator to preserve the original model’s training gradients. It connects gradient mismatch to divergence between LoRA adapters trained on the emulator and the full model, then derives a compression procedure using input and output-gradient statistics, weighted SVD, and adaptive layer-wise regularization. Adapters trained on the emulator can be transferred directly to the full model without post-hoc correction. Experiments on visual question answering, language-model personalization, and image generation demonstrate improvements over EMLoC and robustness to differences between calibration and downstream data.

Critical strengths
  • Empirical breadth: the experiments cover visual question answering, language-model personalization, and image generation.
  • Robustness in task customization: EMLoC suffers a substantial performance drop under this distribution mismatch, with its average accuracy decreasing from 53.3 to 48.4. In contrast, TALE maintains highly comparable performance across both paradigms, achieving 55.9 and 55.3 respectively, and notably outperforms EMLoC by 6.9 points in the server-side setting.
Critical weaknesses
  • For the baseline presented in the paper - the results differ very incrementally, and without std values I’m not sure if the differences are significant. “Consistently outperforms” is too strong.

  • The paper briefly cites OBD-LLM which already derives the same construction: use input and output-gradient statistics to weight both sides of a weight matrix, truncate its SVD, then undo the transformations. Its objective and resulting factors match TALE’s construction, up to notation and the choice of matrix square-root factors. Moreover, preserving adapter gradients when constructing an emulator is already the explicit motivation of GradOT. The combination of these two prior directions makes the broad conceptual and algorithmic novelty claims difficult to accept without a direct comparison.

  • The theoretical analysis provides a local, single-layer justification under simplifying assumptions, but does not directly establish gradient preservation when many layers are compressed simultaneously.

Questions for authors
  • If I understand correctly, Figure 1 reports correlations between output alignment and gradient alignment. However, how does TALE reduce absolute gradient error, or does it preserve gradients throughout adaptation?
5 comments

We are deeply grateful for your detailed review and for engaging with our method at the level of its derivation. Your comments clearly identified where our original claims exceeded the evidence. To address this, we have not only adjusted our statements for precision but also added in-depth analytical evidence, ensuring our contributions are now fully supported.


Significance of Empirical Gains and Claim Calibration (W1)

We have moderated the performance claims in the revised manuscript. To address the variability concern, we conducted an additional three-seed comparison on five benchmarks at 50% compression.

[Table A] Accuracy gains over EMLoC at 50% compression (mean ± std over three seeds; percentage points).

ChartQAInfoVQAPMC-VQAWebSRCWC-VQA
Gain (TALE − EMLoC)+0.83 ± 0.22+2.47 ± 0.35+1.75 ± 0.87+1.65 ± 0.16+1.91 ± 0.43

These results support the gains reported in the paper, with mean improvements exceeding the corresponding across-seed standard deviations on all five benchmarks.

The magnitude of improvement is also meaningful relative to full-model LoRA. The results show that TALE with adaptive regularization reduces the average gap from 2.94 to 1.18 percentage points, closing 59.9% of EMLoC’s gap to full-model performance. We will incorporate these findings into the manuscript and make effort to extend the multi-seed evaluation and mean ± std reporting to all main results.


Clarifying TALE’s Conceptual and Algorithmic Distinctions from Prior Work (W2)

Unlike OBD-LLM, which targets inference fidelity, and GradOT, which maximizes model privacy at the risk of suboptimal downstream adaptation, TALE directly targets training-signal fidelity without such constraints.

  • Contribution beyond OBD-LLM: Despite similar compression formulations, TALE explicitly addresses gradient discrepancies that accumulate across fine-tuning steps (Proposition 1) and propagate across layers through backpropagation, introducing layer-wise adaptive regularization to stabilize empirical gradient statistics (SGS_G). As a result, TALE outperforms OBD-LLM after adaptation, across three domain-specific datasets (Table B) and under server-side distribution shifts (Table C), where ill-conditioned gradient statistics can increase sensitivity to calibration errors and amplify downstream gradient mismatches.

  • Conceptual Distinctions from GradOT: While both methods aim to preserve gradients, GradOT maximizes model privacy by intentionally degrading the compressed model’s standalone performance, introducing a trade-off between privacy and adaptation fidelity. As shown in Table D, GradOT achieves lower downstream performance than TALE at the same compression ratio, despite its gradient-preserving objective.

[Table B] Comparison of TALE with OBD-LLM and EMLoC on three domain-specific downstream adaptation benchmarks.

PMC-VQAWebSRCWC-VQA
Full49.586.853.5
EMLoC48.682.548.9
OBD-LLM46.081.850.6
Ours (Adaptive)49.584.351.2

[Table C] Downstream task customization performance of OBD-LLM and TALE under user-side and server-side settings.

UserServer
Full Model59.5-
EMLoC53.348.4
OBD-LLM55.652.1
Ours (Adaptive)55.955.3

[Table D] Performance comparison of TALE and GradOT at different compression ratios on general-domain (InfoVQA) and domain-specific (WebSRC) benchmarks.

InfoVQAWebSRC
Full Model (100%)47.286.8
GradOT (80%)45.672.1
GradOT (50%)40.073.0
Ours (50%)46.184.3

Gradient preservation under multi-layer compression (W3, Q1)

To examine gradient preservation beyond the local theoretical setting, we added block-wise gradient alignment measurements on the actual compressed model in the new Figure 1(b). Across LLM blocks, TALE achieves roughly twice the average gradient alignment of EMLoC, with the most evident gap emerging in the early blocks where EMLoC's alignment drops near zero. We further extend this analysis in the new Section 4.6 to examine accumulated LoRA updates, transfer losses, and training trajectories, supplemented by step-wise gradient alignment across fine-tuning in Appendix G.


We welcome your feedback on whether our revised claims now accurately reflect the evidence, and will expand our variance and comparative analyses in the final version. We hope the revised manuscript addresses your concerns and invites a more favorable assessment.

Reviewer_9cwYOpenReview ↗

Response to Rebuttal

Thanks for your response.

  • Comparison to the OBD-LLM and GradOT:

    • the rebuttal result looks like authors selected the datasets on which the results are most promissing, otherwise - how is this selection is motivated? I think that the authors should extend the main Table with full results from these two relevant methods.
    • The authors attempt to distinguish TALE by saying that OBD-LLM focuses on inference compression while TALE focuses on gradient preservation during adaptation. But Theorem 1 (Eq. 4) and Corollary 1 (Eqs. 5 - 6) appear to recover essentially the same weighted low-rank approximation objective and closed form solution already derived in OBD-LLM (Section 4.2, Eqs. 9–13). Both methods use input and output-gradient covariance statistics to construct a doubly transformed weight matrix, apply truncated SVD, and invert the transformations to obtain the compressed weights.
  • Accuracy gains over EMLoC at 50% compression: again I dont see all datasets. In addition the most important configuration is 25% + QLora which combines aggressive memory reduction with quantization, a natural setting for resource-constrained adaptation. TALE does provide improvements with fixed regularization, but their magnitude is limited, and the proposed adaptive regularization fails to improve the overall result in this configuration. This provides a concrete counterexample to the manuscript's broad claims of consistent improvement.

  • Regarding your last point:

    • I think there is a type in the response since I don't see Appendix G in the paper.
    • My question was not answered, I asked about absolute gradient error, not correlation.
    • according to the results in the new plot (Figure 5), although these values are consistently higher than EMLoC's, the gradients become nearly orthogonal to those of the full model after the early adaptation stage. This is a substantial limitation for a method claiming to faithfully preserve training signals.
  • Additional points I would like to discuss after an additional review I've made:

    • On-device feasibility is not convincing (Appendix C, Table 6). The method has approximately 2.37×2.37\times the emulator construction compute of EMLoC and provides no additional peak-memory reduction during adaptation.

    • The paper claims robust generalization to unseen tasks and users. However, the task-generalization experiments are conducted across four question categories within the same VQA-Med dataset. This is a relatively restricted setting compared with genuinely unseen datasets or substantially different domains. Moreover, in Table 2, the Abnormality task has extremely low accuracy for every method, including full-model fine-tuning (6.2%). Therefore, that task provides little meaningful evidence of successful adaptation.

Overall, I believe the paper requires substantial revision, including more comprehensive empirical comparisons with the most closely related methods, a more carefully narrowed theoretical contribution, and a detailed discussion of the methodological similarities and differences between TALE, OBD-LLM, and GradOT.

AuthorsOpenReview ↗

Response to Reviewer 9cwY (1/3)

Dear Reviewer 9cwY,

Thank you for the careful second-round review. Your summary names three revisions the paper needs: (i) comprehensive comparisons with OBD-LLM and GradOT, (ii) a more carefully narrowed theoretical contribution, and (iii) a detailed discussion of how TALE, OBD-LLM, and GradOT relate. The revised manuscript makes all three. This response also adds the evidence that our first response lacked:

  • Full baseline comparison: On all seven VQA benchmarks, TALE outperforms GradOT on all seven benchmarks and OBD-LLM (Table E).
  • 25% + QLoRA: With dataset-specific λ, adaptive regularization becomes the best TALE variant and outperforms EMLoC on average (Table F).
  • Absolute gradient error: TALE has lower error than EMLoC in 23 of 24 LLM blocks (Table G, Appendix F).
  • On-device cost: Emulator construction adds 3.5% total compute over EMLoC and stays within inference-level memory.
  • Distribution shift: When calibration and fine-tuning data differ, TALE retains its accuracy while EMLoC degrades (e.g., WebSRC 84.5 vs. 76.9; Table H).

Below, we answer each remaining point directly, including those on which we agree with you.

Comparison with Gradient-Based Baselines and Clarification of TALE’s Contributions

  • Full Comparison with GradOT and OBD-LLM across VQA Benchmarks

We apologize for the partial comparison in our first response. Given the time and resource constraints during the rebuttal period, we initially reported results on datasets with completed experiments. We now provide additional results, which reinforce our original conclusions. Our method outperforms GradOT across all evaluated datasets, with gains of up to 12%p despite more aggressive compression, and surpasses OBD-LLM on six out of seven benchmarks, with particularly pronounced improvements on domain-specific datasets.

[Table E] Full comparison with gradient-based compression baselines across major VQA benchmarks.

MethodChartQADocVQAInfoVQATextVQAPMC-VQAWebSRCWC-VQA
GradOT (80%)82.383.945.679.545.172.148.1
OBD-LLM (50%)85.184.845.279.646.081.850.6
Ours (50%, Adaptive)85.184.946.179.749.584.351.2
  • Clarification on TALE’s Theoretical and Algorithmic Contributions

We further clarify that distinguishing TALE from a training perspective is not merely a conceptual distinction but provides the foundation for our theoretical contributions. Specifically, Proposition 1 establishes that the terminal divergence between the adapters of the full model and the emulator is bounded by the accumulated per-step gradient gaps, theoretically justifying the minimization of LoRA gradient gaps. This motivates our formal definition of a training-aware emulator and the subsequent derivation of Theorem 1. Such training-oriented theoretical justification is absent in OBD-LLM, which approaches emulation from an inference perspective, despite the mathematical equivalence arising from the approximation and decomposition of the LoRA gradient gap. TALE further introduces an additional algorithmic contribution: the adaptive regularization.

  • Empirical and Analytical Contributions

Finally, we introduce server-side and user-side on-device training scenarios under different assumptions regarding computational resources and data access. Through extensive experiments, we demonstrate the effectiveness of TALE in improving adaptation performance in user personalization (Figure2) and task customization (Table2). Furthermore, our in-depth analysis in Figure 4 examines the extent to which and how TALE preserves the full model’s training signals from three complementary perspectives. These empirical and analytical contributions are absent in OBD-LLM.


Reviewer_DPQXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TALE, which trains LoRA adapters on a compressed model and transfers them back to the original model. The main idea is to use gradient information when building the compressed model, so that it provides more useful training signals. The paper evaluates this approach across several adaptation and personalization tasks showing promising efficiency of adaptation.

Critical strengths
  1. The problem is practical. Users may be able to run a large model but lack the memory to fine-tune it, so training adapters on a smaller model is useful.

  2. The motivation makes sense. If an emulator is used for training, preserving how the model learns is a reasonable goal. Incorporating gradient information is a natural way to approach this.

  3. The experiments cover different models and tasks, with promising improvements over EMLoC. Reusing a server-built emulator across users is also a useful deployment setting.

Critical weaknesses

1.The incremental contribution over closely related methods is insufficiently established. OBD-LLM, cited in the paper, already derives a two-sided weighted low-rank approximation using input and output-gradient statistics. GradOT also explicitly studies gradient-preserving emulator construction, although its setting and algorithm differ. The adapter-training interpretation is interesting, but comparisons primarily against EMLoC do not isolate the contribution beyond existing gradient-aware compression. A matched evaluation of OBD-LLM within the same LoRA training and transfer pipeline would be particularly informative. 2. The connection between the theoretical surrogate and actual training dynamics is not sufficiently validated. 3. The reported peak GPU memory is 20.5 GB versus 23.4 GB for full-model LoRA, and no measured inference-memory reference is provided to substantiate the inference-level-budget claim. Server-side reuse can reasonably amortize construction, but separate construction/adaptation memory measurements and a clear accounting of adaptive-table selection would strengthen the deployment argument.

[1] Yuhang Li et al. 2026. Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation. arXiv preprint arXiv:2604.00821. Paper.

[2] Kai Yao et al. 2025. GradOT: Training-free Gradient-preserving Offsite-tuning for Large Language Models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5115–5130. Paper.

Questions for authors

1.What does TALE add over using OBD-LLM to construct the emulator and then following the same LoRA training and transfer procedure?

  1. Can you show directly that LoRA trained on TALE learns updates closer to those learned on the original model?

  2. When is the extra cost of constructing TALE worth paying, especially for a user who only needs to fine-tune once?

  3. How sensitive is TALE to the choice of calibration data, especially when the downstream task differs substantially from the data used to build the emulator?

3 comments

We truly appreciate your careful and constructive review, and in particular for specifying exactly which evidence would make our contribution convincing. Below, we clarify each of your comments in detail.


Clarifying TALE’s Contributions beyond OBD-LLM and Matched Evaluation (W1, Q1)

Unlike OBD-LLM, which targets inference fidelity, TALE directly aims to preserve training signals for downstream adaptation. Despite similar compression formulations, TALE explicitly addresses gradient discrepancies that accumulate across fine-tuning steps (Proposition 1) and propagate across layers through backpropagation, introducing layer-wise adaptive regularization to stabilize empirical gradient statistics (SGS_G). Empirically, we demonstrate that TALE outperforms OBD-LLM after adaptation across three domain-specific datasets (Table A) and under server-side distribution shifts (Table B), where ill-conditioned gradient statistics can increase sensitivity to calibration errors and amplify downstream gradient mismatches.

[Table A] Matched comparison of OBD-LLM and TALE on domain-specific downstream adaptation tasks.

PMC-VQAWebSRCWC-VQA
Full49.586.853.5
EMLoC48.682.548.9
OBD-LLM46.081.850.6
Ours (Adaptive)49.584.351.2

[Table B] Downstream task customization performance of OBD-LLM and TALE under user-side and server-side settings.

UserServer
EMLoC53.348.4
OBD-LLM55.652.1
Ours (Adaptive)55.955.3

Does the theoretical surrogate translate into actual training dynamics? (W2, Q2)

We agree that the theoretical surrogate should be validated against actual training dynamics. The new Section 4.6 therefore compares accumulated LoRA updates, transfer losses, and training trajectories against full-model fine-tuning. We are grateful for this constructive suggestion, as it guided us to provide a much clearer empirical account of how TALE faithfully tracks full-model training dynamics.


Memory measurements and cost accounting for practical deployment (W3, Q3)

During the rebuttal, we optimized TALE's implementation to process the model layer by layer, matching EMLoC's pipeline. We have updated Appendix C accordingly to reflect these improvements. With this optimization, TALE's construction peak memory is within 7% of EMLoC's. Fine-tuning on the emulator peaks at 14.2 GB, compared with 24.9 GB for full-model LoRA, a 43% reduction. We also state in the revised paper that this peak remains below the inference-level memory of the 8B model (16.5 GB).


TALE’s Robustness to Calibration-to-Downstream Distribution Shifts (Q4)

  • Cross-Dataset Adaptation: To evaluate robustness under distribution shifts, we construct an emulator using the general VQA dataset (InfoVQA) and evaluate downstream adaptation on two domain-specific benchmarks (PMC-VQA and WebSRC) from Table 1. As shown in Table C, TALE demonstrates greater robustness than the activation-preserving EMLoC, with further improvements from adaptive damping, which mitigates sensitivity to ill-conditioned gradient statistics and calibration errors.

  • Task Customization under Server-Side Distribution Shifts: Similarly, Our task-customization experiments in Table2 further support this robustness. Under server-side distribution shifts, where calibration data exclude the target task, TALE maintains stable performance (55.9 →\rightarrow 55.3), whereas EMLoC suffers a substantial drop (53.3 →\rightarrow 48.4).

[Table C] Cross-dataset adaptation under calibration-to-downstream distribution shifts. Matched and Shifted indicate whether the calibration and fine-tuning data distributions are the same or different, respectively.

PMC-VQAWebSRC
MatchedShiftedMatchedShifted
EMLoC48.647.182.576.9
Ours48.547.381.881.9
Ours (Adaptive)49.547.884.384.5

We believe the revised paper now provides the matched comparison and direct evidence you identified as missing. If these results resolve your concerns, we would be grateful if you would consider updating your assessment.

Dear author, Thank you for your detailed responses, which have addressed my concerns. I have raised my score accordingly.

Dear Reviewer DPQX,

Thank you for taking the time to review our responses and for raising your score.

We will continue to refine the paper beyond the rebuttal period to make it as robust and complete as possible. We sincerely appreciate your thoughtful engagement throughout the review process.

Best regards,
Authors

Reviewer_TW2gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents TALE, a method that improves the construction of an emulated model to better suit the needs of on-device training. Traditional emulation construction techniques aim to minimize the divergence of the outputs of the emulated model and the original model. However, similar outputs are not a guarantee of similar gradients, meaning the proxy model may end up with very different local adapters after fine-tuning that are not representative of the adapters that would have been trained under the original model's gradient updates. TALE addresses this problem by constructing an emulator from SGS_G and SXS_X which are constructed from the full-model gradient on a single batch and the second moment of the input, respectively. TALE improves over EMLoC in a variety of settings, demonstrating that emulation construction should be gradient, as well as output, aware.

Critical strengths
  1. Writing and Motivation - The writing and motivation of the paper is generally clear. Figure 1, Section 3.2, and Section 3.3, are especially effective in conveying the core idea. The method and reasoning are outlined well, and most ideas extend naturally from the claims the paper makes.
  2. Broad Benchmarking - The paper demonstrates the effectiveness of the idea on different baselines, different budgets, and different tasks (even including diffusion models in the results). The variety of benchmarking results are appreciated and useful in showing the generality of the method.
Critical weaknesses
  1. Lack of Representative Baselines - In the Related Works section, the paper mentions two other gradient-aware emulator construction methods, namely OBD-LLM and GradOT. Looking at these methods, they seem to share relevant characteristics with the proposed work, especially in the case of OBD-LLM. It is unclear why these methods are not included in the baseline comparison of TALE, unless I am misunderstanding why the methods would not also apply in the settings being tested. More clearly differentiating TALE from prior works and showing that it improves on related methods would be very helpful in solidifying the claims of the paper.
  2. Somewhat Marginal Main Results - The main results table compares to EMLoC, but often shows only marginal improvements over the method. In particular, it occasionally relies on one of the lambda settings (0, 1, adaptive) to be better than the baseline while the others are sometimes worse. Since adaptive does not universally perform better than 0 or 1, it's unclear how choice of lambda impacts the performance of the method and whether we should read this table as the method being consistently better than EMLoC or closer to on-par.
  3. Missing Ablation - It would be important to see how much the gradients diverge from the full model gradients over multiple training iterations. Right now, it makes sense that the construction of the emulator may align gradients better at the start of training by design, but whether gradients remain close to the original model's gradients as the adapters evolve. I'm not sure if Figure 1 establishes the consistency.
Questions for authors
  1. TALE calculates the full-model gradient in order to construct the emulator. Section 4.3 addresses this requirement through server-side construction, but how is this step performed in the user-side construction setting with device memory constraints? It would be useful to clarify when local construction is feasible and when server-side construction is necessary.
3 comments

We thank the reviewer for the encouraging feedback and recognition of TALE's motivation, theoretical reasoning, and extensive validation. We address each point below.


Extensive Empirical Comparison of TALE with Prior Gradient-Aware Methods (W1)

To address this concern, we conducted additional experiments comparing TALE with two related baselines, GradOT and OBD-LLM.

  • Comparison with GradOT: While both methods aim to preserve gradients, GradOT maximizes model privacy by intentionally degrading the compressed model’s standalone performance and employs selective channel pruning for MLP layers. As shown in Table A, TALE achieves superior downstream adaptation performance at the same compression ratio, despite GradOT’s gradient-preserving objective.

  • Comparison with OBD-LLM: Unlike OBD-LLM, TALE explicitly considers gradient discrepancies that accumulate across fine-tuning steps (Proposition 1) and propagate across layers through backpropagation, introducing layer-wise adaptive regularization to stabilize empirical gradient statistics (SGS_G). Empirically, we demonstrate that TALE outperforms OBD-LLM after adaptation across three domain-specific datasets (Table B) and under server-side distribution shifts (Table C), where ill-conditioned gradient statistics can increase sensitivity to calibration errors and amplify downstream gradient mismatches.

[Table A] Comparison of TALE and GradOT at different compression ratios on general-domain (InfoVQA) and domain-specific (WebSRC) benchmarks.

InfoVQAWebSRC
Full (100%)47.286.8
GradOT (80%)45.672.1
GradOT (50%)40.073.0
Ours (50%)46.184.3

[Table B] Comparison of TALE with OBD-LLM and EMLoC on three domain-specific downstream adaptation benchmarks.

PMC-VQAWebSRCWC-VQA
Full49.586.853.5
EMLoC48.682.548.9
OBD-LLM46.081.850.6
Ours (Adaptive)49.584.351.2

[Table C] Comparison of TALE with OBD-LLM and EMLoC under user- and server-side emulator construction for downstream task customization.

UserServer
EMLoC53.348.4
OBD-LLM55.652.1
Ours (Adaptive)55.955.3

Clarifying TALE’s gains over EMLoC and the role of adaptive (W2)

We agree with the reviewer that the main table contains exceptions and that adaptive regularization does not always outperform the unregularized variant (λ\lambda = 0). During the rebuttal, we further examined when adaptive regularization matters. The additional VQA-Med-2019 results below show that its benefits are particularly pronounced in the server-side setting. We are investigating the underlying reasons and will share our analysis once it is complete.

[Table D] Effect of adaptive regularization on VQA-Med-2019. The benefits of adaptive regularization over the unregularized variant are more pronounced when server-side calibration data differ from downstream fine-tuning data.

Reg.ModalityPlaneOrgan
User (S&T)None69.073.672.8
Adaptive72.873.874.6
Server (S&T)None62.871.666.6
Adaptive72.674.871.6

Does TALE preserve gradient alignment throughout training? (W3)

During the rebuttal, we added extensive analyses to Section 4.6 and a direct step-wise gradient comparison to Appendix G. TALE maintains higher gradient alignment with the full model than EMLoC throughout all 100 training steps, with a 1.9×1.9\times advantage on average across five calibration draws.


When Is Local Emulator Construction Feasible, and When Is Server-Side Construction Needed? (Q1)

The feasibility of local construction depends on the available device memory. Although TALE requires full-model gradients, its memory-efficient statistics collection avoids optimizer states and large activation buffers, enabling construction within an inference-level memory budget.

1. When local construction is feasible:

  • When the device has sufficient memory to support full-model inference.
  • As shown in Table 6, TALE construction requires only 9.5 GB, below the original model's 16.5 GB inference footprint and the 24.9 GB required for full-model LoRA fine-tuning.

2. When server-side construction is necessary:

  • Server-side construction is particularly beneficial for multiple users or downstream tasks, as it eliminates the overhead of repeated local emulator construction.
  • In the OPPU personalization setting (Figure 2), user-side construction requires a separate emulator for each of the 20 users, whereas server-side construction enables a single emulator to be shared across all users while maintaining comparable or even superior performance.

Your insightful suggestions provide clear directions for improvement, and we will actively incorporate them into the paper.

Thank you for your thorough response. While I agree with much of what you have mentioned, the marginal results are still slightly concerning to me. In particular the additional ablation is on VQA-Med-2019, and based on Table 2, using λ=0\lambda=0 can actually sometimes underperform relative to EMLOC. In contrast, Table 1 contains settings where adaptive regularization underperforms relative to λ=0\lambda=0 or λ=1\lambda=1. This means there's not much clarity about the best strategy for setting λ\lambda so that there are consistent gains over prior baselines. That being said, I do find the paper useful and well-written as a whole. For that reason, I will be maintaining my, already positive, score of a 3.

Dear Reviewer TW2g,

Thank you very much for your thoughtful engagement and for finding the paper useful and well-written. We sincerely appreciate your positive assessment and your continued support. Below, we address your remaining concern about how to set λ\lambda to obtain consistent gains over prior baselines.

λ\lambda Selection Strategies

In Table 1, adaptive regularization uses a single λ\lambda table that is shared across all datasets and settings without re-tuning. As a result, it does not fully show the effect of adaptive regularization. We tested this in the 25% + QLoRA setting, where TALE's gains over EMLoC were most limited.

[Table A] Accuracy at 25% + QLoRA with shared and dataset-specific adaptive λ\lambda (%).

25% + QLoRAλ\lambdaChartQADocVQAInfoVQATextVQAPMC-VQAWebSRCWC-VQAAvg.
EMLoC–79.484.140.179.545.172.946.764.0
TALE (Table 1)Adaptive (shared)79.384.540.079.443.674.047.364.0
TALE (new)Adaptive (per-dataset)81.284.640.079.545.774.247.564.7

With dataset-specific λ\lambda, adaptive regularization achieves the highest average accuracy (64.7), above EMLoC and every TALE variant in Table 1. It also matches or exceeds the shared table on every dataset. After the rebuttal phase, we will apply this dataset-specific adaptive regularization to all configurations in Table 1 and update the paper accordingly.

Your comments helped us clarify both the strengths and the limits of TALE, and we are truly grateful for your support of this work.

Best regards,
The Authors

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your thoughtful feedback and suggestions. We have uploaded a revised version of the paper that refines the wording and updates the computational cost measurements after optimizing the implementation. The revision also includes five new analyses:

  1. Block-wise gradient alignment (Introduction, Figure 1b)
  2. Accumulated LoRA updates (Section 4.6, Figure 4a)
  3. Transfer losses (Section 4.6, Figure 4b)
  4. Training trajectories (Section 4.6, Figure 4c)
  5. Step-wise gradients (Appendix G, Figure 5)

We hope the revised paper and our detailed responses help address your questions and clarify the contributions of our work. We remain open to any additional questions or feedback, and we are committed to addressing them to further strengthen our manuscript.

Best regards,
Authors

AuthorsOpenReview ↗

Summary of Responses and Revisions

We sincerely thank all three reviewers for their careful reading and constructive feedback. Their critiques prompted a substantial round of new experiments and analyses, and helped us position TALE more precisely relative to prior work.

During the discussion, Reviewer DPQX indicated that our responses addressed their concerns and raised their score, and Reviewer TW2g maintained their positive assessment, finding the paper useful and well-written. Reviewer 9cwY's follow-up identified three revisions the paper needs; our second-round response answers each point, and all three are now reflected in the revised manuscript. As the discussion period draws to a close, we consolidate our responses and the resulting revisions here.


Strengths Recognized by the Reviewers

The reviewers credit the following in particular:

(1) A practical problem with a well-founded motivation: fine-tuning often exceeds the on-device memory budget, and an emulator used for training should preserve how the model learns, not only what it outputs (DPQX S1, S2; TW2g S1).

(2) Broad empirical validation: multimodal VQA, LLM personalization, and diffusion-based image generation, under multiple memory budgets (TW2g S2; DPQX S3; 9cwY S1).

(3) Robust, reusable server-side construction: in task customization, TALE remains stable when the calibration data exclude the target task, whereas EMLoC degrades substantially, and a single server-built emulator can be shared across users (9cwY S2; DPQX S3).


Common Concerns and How They Are Addressed

C1. Relation to OBD-LLM and GradOT (TW2g W1; DPQX W1, Q1; 9cwY W2). In the matched comparison requested by Reviewer DPQX (Q1), we compared TALE and OBD-LLM using the same LoRA training and transfer pipeline. TALE matches or exceeds OBD-LLM across all seven VQA benchmarks, with clear gains on domain-specific datasets. It also outperforms OBD-LLM by approximately three percentage points in server-side task customization. TALE further outperforms GradOT on all seven benchmarks despite using a smaller emulator budget. Following our initial response, Reviewer DPQX indicated that their concerns had been addressed. While TALE’s weight-space objective coincides with the K-FAC objective used by OBD-LLM (Section 3.3), the two methods motivate it from different perspectives. TALE provides a training-oriented justification: under the stated assumptions, Proposition 1 bounds adapter drift in terms of accumulated gradient discrepancies, motivating the objective used for emulator construction. TALE also introduces objective-based, layer-wise adaptive regularization, whose benefits are particularly pronounced in the evaluated server-side setting. We provide a detailed comparison with OBD-LLM and GradOT in Appendix G.

C2. Magnitude of gains and the choice of λ\lambda (TW2g W2; 9cwY W1). We have moderated the claims of consistent improvement in the revision. A three-seed study confirms that TALE's mean gains over EMLoC exceed seed-to-seed variation on all five benchmarks where we claim gains, and on average TALE closes about 60% of EMLoC's gap to full-model LoRA. Adaptive regularization is particularly beneficial when calibration and fine-tuning distributions differ, as confirmed in both server-side task customization and cross-dataset calibration experiments. Its gains in Table 1 are understated because a single λ\lambda table was shared across all settings; selecting the table per dataset with the same objective-based procedure makes it the best TALE variant.

C3. Training-signal preservation across layers and throughout training (TW2g W3; DPQX W2, Q2; 9cwY W3, Q1). We expanded the analysis to evaluate training fidelity through block-wise gradient alignment and gradient error (Introduction and Appendix F), gradient alignment throughout adaptation (Appendix E), and accumulated LoRA updates, transfer losses, and training trajectories (Section 4.6). Together, these analyses show that TALE more closely approximates the full model’s training signals than EMLoC and produces adapters that transfer more effectively to the full model in the evaluated setting.

C4. On-device cost and feasibility (TW2g Q1; DPQX W3, Q3; 9cwY follow-up). With a layer-by-layer implementation, both emulator construction and fine-tuning stay below the original model's inference memory, unlike full-model LoRA. Construction is a one-time step, so TALE adds only 3.5% total compute over EMLoC at the same emulator size, and it can also move to the server, where one shared emulator matches or outperforms per-user EMLoC emulators built on-device.

We are deeply grateful to the reviewers for guiding these improvements. We are committed to continuing this revision, extending per-dataset λ\lambda selection and multi-seed reporting to all main configurations in the final version. We hope the revised manuscript conveys this contribution clearly and addresses the reviewers' concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.