acceptodds
Under review as a conference paper at ICLR 2027

Instant Context Optimization: Compiling Few-Shot In-Context Demonstrations into Pre-Trained Channel Magnitudes via Circuit-Bounded Sparse Optimization

Abstract

In-Context Learning (ICL) enables Large Language Models (LLMs) to perform tasks from a small set of demonstrations (e.g., ), but requires persistent Key-Value (KV) cache expansion and quadratic prefill latency on every forward pass. While compiling demonstrations directly into model weights would eliminate this runtime prompt tax, standard few-shot parameter optimization () is fundamentally ill-posed, causing severe overfitting and representation collapse. In this work, we propose Instant Context Optimization (ICO), a framework that compiles demonstrations into task-specific static weights once per task offline, recovering the performance benefits of ICL without serving-time prompt costs. Formally framed as episodic prompt-to-weight internalization via circuit-bounded test-time distillation, ICO leverages three structural constraints exposed by the model's own in-context activation routing: (1) Where to update: Gradient-based parameter saliency localizes updates to sparse, task-relevant subcircuits (); (2) How to update: Updates are restricted to channel-wise gain modulation along fixed directional bases (), provably preserving the layerwise kernel nullspace (); and (3) How to score updates: Reasoning trajectories are supervised via Demonstration-Conditioned Teacher Reference Scoring with Group Minimum-Anchored Relative Advantage (). Across 38 benchmarks on models up to 27B parameters, ICO recovers (up to on GSM8K) few-shot headroom with a universal grand macro served gain in pure zero-shot setting, safeguarded by an empirical Risk-Monotonic AR validation gate.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test-time Training, Efficient Method

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2Y81OpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Instant Context Optimization (ICO), which compiles KK in-context demonstrations into weights offline so the model can be served zero-shot. Each linear projection is reparameterized as W∗=diag(m/m0) W0W^* = \mathrm{diag}(\mathbf{m}/\mathbf{m}_0)\,W_0, so only per-output-channel gains are trained (~1.4M parameters on Llama-3-8B). The authors report +11.25 points over base on a 9-task development suite (Llama-3-8B), +12.67 points macro over 38 tasks × 4 models (8B–27B), and serving-efficiency gains over ICL.

Critical strengths

The problem is practical and well motivated.

Critical weaknesses

The reported numbers are internally inconsistent, and some cannot be produced by the described procedure.

  • AR (Eq. 8) either accepts the candidate or reverts to W0W_0, so each "Gold SFT (+AR)" entry in Table 2 should match either the Gold SFT entry or the Base entry. 8 of 9 match neither. For example, mbpp is 39.80 (SFT) and 40.80 (base), yet 30.80 with AR.
  • In Table 8, different models share identical results. Base GSM8K is 41.09 for three models, and the teacher GSM8K score is 53.68 for all four. The headline 107.8% headroom is computed against that shared teacher value.

Because of these inconsistencies, I cannot assess the empirical claims.

Questions for authors
  1. Please explain why several base and teacher numbers in Table 8 are identical across different models.

  2. Please report LoRA, (IA)3^3, and all-linear gain SFT trained with the same LOO rollout-distillation objective and the same rollback gate as ICO.

3 comments
AuthorsOpenReview ↗

Thank you for actionable suggestions.

1. Question 1: Identical Base and Teacher Numbers Across Models in v1 Table 8

As documented in Row 1 above, we used unsloth/gemma-* and google/gemma-* interchangeably across two machines, which caused the scripts to fall back to Llama-3-8B defaults (41.09% / 53.68% on GSM8K, 11.33% on MGSM, 31.34% / 63.21% on BBH, 69.05% / 68.62% on LongBench; note that on MBPP 0-shot, Gemma-2-9B genuinely scores 204/500 = 40.80%, matching Llama-3-8B, whereas its 3-shot Teacher is 53.80% vs. 49.80%). Dividing Gemma-2-9B's authentic 54.66% GSM8K score by Llama-3-8B's 53.68% Teacher produced the erroneous 107.8% headroom. Below are the corrected GSM8K and 38-task macro numbers across the 4 core models (Table 3, Figure 3b, full 5-scenario ×\times 7-model breakdown in Appendix Table 16):

ModelGSM8K Base / TeacherGSM8K ICO (Headroom)38-Task Base / Teacher38-Task Compiled / Served
Meta-Llama-3-8B41.09% / 53.68%50.64% (75.9%)31.66% / 61.89%38.98% / 40.31%
Gemma-2-9B21.38% / 68.61%54.66% (70.5%)38.19% / 67.78%41.64% / 47.37%
Gemma-3-12B-PT40.26% / 74.07%59.74% (57.6%)40.66% / 67.31%47.92% / 50.23%
Gemma-3-27B-PT63.00% / 80.82%71.57% (48.1%)48.66% / 74.74%53.02% / 55.04%

2. Question 2: LoRA, (IA)³, and All-Linear Gain SFT Under Identical LOO Rollout Distillation and Rollback Gate

We trained LoRA (r=8r=8), (IA)^3, unregularized All-Linear Gain + LOO (τ0=0\tau_0=0, no soft thresholding or TIES), and recent PEFT methods on Meta-Llama-3-8B with the same LOO rollout-distillation objective (S=8S=8, diff_bot, λgold=2.0\lambda_{\text{gold}}=2.0) and rollback gate (+AR) (Table 2, Figure 3a, scenario and per-task scores in Appendix Tables 11, 13):

Method (Meta-Llama-3-8B, Same LOO + AR)Params (BF16)9-Task Compiled / +AR38-Task Compiled / +AR
Base Zero-Shot / KK-Shot Teacher039.43% (Base) / 62.26% (ICL)31.66% (Base) / 61.89% (ICL)
Direct SFT 1D-Gain (S=0,τ0=0S=0, \tau_0=0, no LOO)1.38M (2.75 MB)41.99% / 42.32%—
LoRA (r=8r=8) + LOO + AR20.97M (41.94 MB)29.07% / 43.22%37.23% / 41.80%
(IA)^3 + LOO + AR0.52M (1.05 MB)43.01% / 48.05%38.07% / 39.43%
All-Linear Gain + LOO (τ0=0\tau_0=0, no TIES)1.38M (2.75 MB)46.16% / 51.54%—
ICO (τ0=0.10\tau_0=0.10 + TIES, Ours)1.38M (2.75 MB)45.86% / 50.68%38.98% / 40.31%

Core Findings (§ 4.2):

  1. 1D all-linear channel scaling vs. LoRA and (IA)^3: Under identical LOO supervision and rollback gating, both 1D channel-gain variants (51.54% and 50.68%) beat LoRA (43.22%) and (IA)^3 (48.05%) on the 9-task suite, avoiding LoRA's severe raw collapse on few-shot math/code (29.07% Raw). Across all 38 tasks, ICO achieves the highest raw Compiled accuracy (38.98%), while LoRA + AR reaches 41.80% (via rank-8 capacity on symbolic BBH tasks once collapsed runs revert to Base) at 15.2×15.2\times larger adapter memory.
  2. Effect of LOO distillation vs. sparsity: LOO rollout distillation contributes +8.36% to +9.22% over Direct SFT 1D (41.99% -> 42.32%). When S=8S=8 rollouts provide dense supervision, unregularized τ0=0\tau_0=0 slightly increases multi-round plasticity (51.54% vs. 50.68%), whereas τ0=0.10\tau_0=0.10 curbs raw drift on boolean/cross-lingual tasks, improves S=0S=0 compilation (+3.19%, Appendix Table 12), and confines updates to ≤10%\le 10\% of channels. Both are compared side-by-side in Table 2 and Appendix Table 13.

I think these inconsistencies show the manuscript is under-polished and I recommend the authors to resubmit to other late venues.

Thank you for your suggestions. We appreciate that you have identified a critical bug in our paper and experiment post-processing.

Reviewer_4bEgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ICO converts a small set of in-context demonstrations into channel-wise gain parameters, enabling demonstration-free inference through saliency probing, trajectory supervision, and rollback gating.

Critical strengths

The motivation for demonstration-free inference is practically relevant, and the paper evaluates both inference efficiency and task performance.

Critical weaknesses
  1. All four models used in the main experiments were released in 2024–2025. Including models released in 2026 would strengthen the empirical evaluation and help establish the method’s applicability to more recent models.

  2. The primary baselines, LoRA and (IA)³, both date back to 2022. For an ICLR 2027 submission, comparisons against these methods alone are insufficient to establish competitiveness with recent approaches. I recommend including methods introduced within the past two years.

  3. The authors use Leave-One-Out (LOO) to generate candidate reasoning traces and assign rewards based on the difference in log probabilities with and without demonstrations. However, this reward measures the extent to which the demonstration context supports a candidate answer, rather than the correctness of the answer itself. Consequently, an incorrect reasoning trace may still receive a high weight if it is strongly supported by the demonstrations.

  4. Theorem 1 establishes that scaling by a positive diagonal matrix preserves the null space of an individual linear projection. However, null-space preservation does not imply preservation of the model’s functional behavior. Even if the null space of each linear layer remains unchanged, interactions with attention, nonlinear activations, and residual connections may still lead to substantial changes in the model’s outputs.

  5. The methodology section would benefit from an overview diagram illustrating the complete workflow. This would help readers understand how the individual components fit together and improve the clarity of the presentation.

Questions for authors
  1. Could the authors add at least one model released in 2026 and two to three baseline methods introduced in 2026? For an ICLR 2027 submission, evaluation on current models and comparisons against recent methods are essential to establishing the approach’s relevance and competitiveness.

  2. In Appendix Table 5, adding AR decreases Raw Accuracy from 48.95% to 45.86%. Could the authors explain this degradation and clarify why AR negatively affects performance in this setting?

2 comments
AuthorsOpenReview ↗

Thanks for your actionable suggestions.

1. Weaknesses 1 & 2 and Question 1: Gemma-4-E4B and Recent PEFT Baselines for rebuttal revisions.

  • google/gemma-4-E4B (released in 2026; § 4.4, Table 3 & Appendix Tables 13, 16–17): Across the 9-task 5-category suite without per-task tuning (η=2×10−4,τ0=0.10,Tmax⁡=2,E=5,S=8\eta = 2 \times 10^{-4}, \tau_0 = 0.10, T_{\max}=2, E=5, S=8; Appendix Table 10), ICO improves zero-shot accuracy from 41.70% (Base) to 48.19% Compiled →\to 48.32% Served (+6.62%, 31.4% headroom over the 62.81% teacher, p=3.13×10−2p = 3.13 \times 10^{-2}), including +23.53% on bbh_causal_judgement and +10.08% on gsm8k (expanding the full evaluation to 7 models / 237 runs from 4B to 32B).

  • Recent Baselines (VeRA, DoRA, AlphaEdit-LoRA, plus 2-round SimPO/GRPO/DPO and RIM-10% masking; § 4.2, § 4.3, Table 2, Figure 2a & Appendix Tables 6, 13): Under the identical LOO distillation + AR pipeline on Meta-Llama-3-8B (9 tasks):

    • LoRA (r=8r=8, 20.97M20.97\text{M}): 29.07% -> 43.22% | (IA)^3 (0.52M0.52\text{M}): 43.01% -> 48.05%
    • VeRA (r=256r=256, 1.43M1.43\text{M}): 44.35% -> 44.90%
    • AlphaEdit-LoRA (r=8r=8, 20.97M20.97\text{M}): 44.22% -> 47.07%
    • DoRA (r=8r=8, 22.35M22.35\text{M}): 45.09% -> 49.09%
    • ICO (τ0=0.10\tau_0=0.10 + TIES, 1.38M1.38\text{M}): 45.86% -> 50.68% (and 46.16% -> 51.54% at τ0=0\tau_0=0)

    Finding: Nullspace projection (AlphaEdit-LoRA) and magnitude-direction decoupling (DoRA) recover +15.1%+15.1\% to +16.0%+16.0\% raw accuracy over standard LoRA, independently corroborating our subspace/magnitude hypothesis, while ICO outperforms both at 15.2×15.2\times–16.2×16.2\times smaller BF16 footprint. Unlike prompt-hypernetwork compilers (e.g., Generative Adapters, Chen et al., 2025) that require multi-task meta-training corpora or token-level weight patches (Goldwaser et al., 2026), ICO compiles directly from an off-the-shelf model and K≤8K \le 8 examples (§ 5).

2. Weakness 3: Leave-One-Out Log-Probability Lift vs. Answer Correctness

We clarify in § 3.2, § 4.5, § 7, and Appendix § B.2 that rt(y)r_t(y) measures contextual support from DLOO(k)\mathcal{D}_{\text{LOO}}^{(k)} rather than exact correctness, because programmatic verifiers are unavailable on general NLP tasks and sparse binary rewards vanish when all S=8S=8 rollouts miss hard problems. ICO mitigates supported-but-incorrect traces via three mechanisms: (1) anchoring every update on ground-truth ykgoldy_k^{\text{gold}} with weight λgold=2.0\lambda_{\text{gold}}=2.0, (2) pruning prompt-specific sign conflicts via q0.80q_{0.80} trimming and Vector-TIES sign consensus (§ 3.3), and (3) rejecting degraded checkpoints via held-out validation loss and logit margin Mval\mathcal{M}_{\text{val}} (Eq. 8), with teacher lift ΔICL\Delta_{\text{ICL}} governing rollout signal-to-noise (Figure 4b) and optional compile-time ast.parse() filtering for small code models (§ 4.5, Appendix § B.2).

3. Weaknesses 4 & 5 and Question 2: Multi-Layer Theory, Workflow Overview, and v1 Appendix Table 5

  • W4 (Multi-layer functional behavior): Addressed via Theorem 1(b) (§ 2.2, Eq. 3; Appendix § A.4, Lemmas A.1–A.4), bounding LL-layer non-linear hidden-state shift and output KL divergence across RMSNorm, attention, SwiGLU, and residual connections.
  • W5 (Workflow overview in Methodology): Added concise end-to-end pseudocode with explicit [Phase 1]–[Phase 3] stages in Algorithm 1 (§ 3, page 5) alongside Figure 1 (§ 1, page 2) and Table 1 (§ 1, page 3) showing how Phase 1 (LOO saliency), Phase 2 (LOO rollout distillation + SparseAdamW), and Phase 3 (TIES merging + validation rollback + zero-overhead folding) fit together.
  • Q2 (Why v1 Appendix Table 5 dropped from 48.95% to 45.86% Raw with AR): AR never modifies the raw checkpoint; the drop in v1 came from copy-pasting Rows 1–5 from an un-gated Ktrain=KK_{\text{train}}=K run (48.95%) next to Row 6 from the held-out Ktrain=K−1,Kval=1K_{\text{train}}=K-1, K_{\text{val}}=1 split (45.86% -> 50.68%). In revised Appendix Table 12 (Figure 2b), Rows 5 and 6 share the exact same Ktrain=K−1K_{\text{train}}=K-1 checkpoint (45.86% Compiled →\to 50.68% Served, +4.82% fallback gain).

Thank you for the clarifications and additional experiments. The expanded evaluation and corrected AR comparison address several of my concerns. However, two points remain: (1) please provide evidence that higher log-probability lift correlates with answer correctness, ideally with a uniform-weighting baseline; and (2) please clarify whether the new theoretical bound is informative at the gain magnitudes used in practice, as bounded output changes do not necessarily imply functional preservation.

Reviewer_EmgsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a framework that compiles demonstrations into task-specific static weights once per task offline, based on the authors' observations of three structural constraints revealed by the model's in-context activation routing.

Critical strengths

The research question itself that the paper aims to address is meaningful.

Critical weaknesses

1. The paper is very hard to read, and many important concepts and claims are not sufficiently explained. I list only a few issues that I found even in the first two pages:

  • There are too many mathematical notations (like D, m, r, sigma) introduced in the abstract without proper definitions. Some of these notations are still not clearly explained in the introduction.
  • The term "few-shot headroom" is introduced without any explanation. As far as I know, this is not a standard metric.
  • In line 43, the authors strongly claim that "classical fine-tuning under extreme data scarcity is fundamentally ill-posed" without providing a supporting citation or sufficient justification at that point.
  • The text in Table 1 is too small to read comfortably, especially in a printed copy.
  • What is diff_bot in line 81? The cited paper by Shao et al. (2024) does not use the term "Group Minimum-Anchored Relative Advantages."
  • It is difficult to understand what "Provably Bounded" means in the last entry of Table 1. What exactly is being bounded?

These are only a few examples. I found the paper difficult to follow, even after considerable effort. The presentation makes it difficult to assess the technical contributions and the validity of the claims.

2. The theoretical result in Theorem 1 is trivial and does not provide meaningful theoretical insight.

The authors parameterize the adapted weights as W∗=DW0W^* = DW_0. This directly implies the conclusion in line 149, which follows from elementary linear algebra.

Moreover, this result is purely a consequence of the chosen parameterization and is independent of the optimization procedure. It does not establish any guarantees regarding optimization, generalization, or the prevention of representation collapse. Therefore, I find it difficult to see what meaningful theoretical contribution Theorem 1 provides.

Questions for authors

1. Why is the problem fundamentally ill-posed under extreme data scarcity?

In line 112, the authors claim that compiling K≤8K\leq 8 demonstrations into adapted weights is severely ill-posed when the ambient dimension vastly exceeds the sample size D/K≥106D/K \geq 10^6. Where does this number come from? More importantly, why does this condition imply that the problem is fundamentally ill-posed? Could the authors provide a high-level explanation of the underlying assumptions?

2. Why are binary masks considered brittle?

In line 125, the authors state that "rather than applying brittle binary masks, αl,i\alpha_{l,i} defines a continuous saliency-inverse proximal threshold." What exactly makes binary masks brittle in this setting, and why is the proposed continuous threshold preferable?

3 comments
AuthorsOpenReview ↗

Thanks for your actionable suggestions

1. Weakness 1: Presentation, Notation, Metrics, Citations, diff_bot, and Table 1

  • Zero-symbol Abstract & defined notation: Removed all math symbols (D, m, r, sigma) from the Abstract and defined every variable in words on first use in § 1–§ 3.
  • Few-shot headroom: Defined at first mention in the Abstract ("the accuracy gap between the zero-shot base model and the few-shot prompted model") and formalized in § 4.1.
  • Citations for K≪DK \ll D fine-tuning: Grounded in § 1–§ 2 (kumar2022fine, liu2022fewshot, wainwright2019high) via the rank-≤K\le K empirical loss Jacobian Φ∈RK×D\Phi \in \mathbb{R}^{K \times D}.
  • diff_bot vs. Shao et al. (2024): Clarified in § 1, § 3.2, and § 4.3 that diff_bot (A^k,i=(ri−min⁡jrj)/σk≥0\hat{A}_{k,i} = (r_i - \min_j r_j)/\sigma_k \ge 0) is our minimum-baseline formulation, contrasting it against Shao et al. (2024)'s mean-centered GRPO advantage, which assigns negative weights to half of a small S=8S=8 rollout group and degrades accuracy (Figure 2c, Appendix Table 15).
  • Table 1 legibility & "Provably Bounded": Formatted Table 1 (page 3) at native \footnotesize without \resizebox, and replaced "Provably Bounded" with the explicit quantities: ker(W_0) invariant (Prop. 1); l2 O(eta * T_max * sqrt(d_eff)), KL O(eta^2 * T_max^2 * d_eff) (Thm. 1) (with a symbol key in the caption).

2. Weakness 2: Triviality of v1 Theorem 1

We agree that ker⁡(diag⁡(m⊘m0)W0)≡ker⁡(W0)\operatorname{ker}(\operatorname{diag}(\mathbf{m} \oslash \mathbf{m}_0) W_0) \equiv \operatorname{ker}(W_0) is an elementary linear-algebraic property of parameterization rather than an optimization or multi-layer guarantee. We demoted it to Proposition 1 and proved a substantive two-part Theorem 1 (§ 2.2, Appendix § A.3–A.4, Lemmas A.1–A.4):

  1. Theorem 1(a) (Optimizer Dynamics): Under zero-mean gradient noise on inactive channels, dense AdamW normalization cancels noise scale and induces Θ(ηDgain)\Theta(\eta\sqrt{D_{\text{gain}}}) parameter drift, whereas SparseAdamW (soft thresholding + momentum reset) keeps inactive channels with ∣gt,i∣<12τl,i|g_{t,i}| < \frac{1}{2}\tau_{l,i} at an exact fixed point (ml,i(t)≡m0,l,im_{l,i}^{(t)} \equiv m_{0,l,i}), confining updates to deff≤0.10Dgaind_{\text{eff}} \le 0.10 D_{\text{gain}} (Eq. 2).
  2. Theorem 1(b) (LL-Layer Non-Linear Perturbation): Composing Lipschitz bounds across pre-norm RMSNorm, GQA Softmax attention, SwiGLU MLPs, and residual streams bounds Layer-LL max-token hidden shift ∥h(L)(m∗)−h(L)(m0)∥2,∞\lVert h^{(L)}(\mathbf{m}^*) - h^{(L)}(\mathbf{m}_0) \rVert_{2,\infty} by O(ηTmax⁡deff)\mathcal{O}(\eta T_{\max}\sqrt{d_{\mathrm{eff}}}) (Eq. 3) and next-token KL divergence by O(η2Tmax⁡2deff)\mathcal{O}(\eta^2 T_{\max}^2 d_{\mathrm{eff}}), tightening worst-case ℓ2\ell_2 drift by deff/Dgain≤0.316\sqrt{d_{\mathrm{eff}}/D_{\mathrm{gain}}} \le 0.316 (0.10×0.10\times in KL) over dense updates.

3. Questions 1 & 2: K≪DK \ll D Ill-Posedness and Soft Thresholding vs. Binary Masks

  • Q1 (K≪DK \ll D & slog⁡(D/s)s \log(D/s) assumptions, § 2 & Appendix § A.1): Around θ0∈RD\theta_0 \in \mathbb{R}^D, KK demonstration losses linearize via Φ∈RK×D\Phi \in \mathbb{R}^{K \times D} with rank⁡(Φ)≤K≤8\operatorname{rank}(\Phi) \le K \le 8, leaving a nullspace of dimension ≥D−K\ge D - K where updates fit demonstrations to first order (ΦΔθ⊥=0\Phi \Delta\theta_\perp = \mathbf{0}) while perturbing unseen test inputs. The slog⁡(D/s)s \log(D/s) rate stems from sparse linear recovery under Restricted Isometry Properties (Wainwright, 2019); we explicitly note in § 2 and Appendix § A.1 that non-linear Transformers do not satisfy RIP and use log⁡(Ds)\log\binom{D}{s} only to explain why uninformed support search fails at K≤8K \le 8, motivating KK-shot saliency αl\boldsymbol{\alpha}_l as an informed prior.
  • Q2 (Why binary masks underperform soft thresholding, § 2.1, Figure 2a & Appendix Table 6): (i) Fixing a step-0 binary top-10%10\% mask (RIM-10%) permanently freezes 90%90\% of channels, starving secondary channels recruited during multi-step rollouts and reducing 4-task BBH accuracy by 17.49 points (47.10% Soft ICO vs. 29.61% RIM-10%; Appendix Table 6); (ii) dynamic hard top-kk jumps discontinuously at rank ties, destabilizing Adam moments; (iii) weighted ℓ1\ell_1 soft thresholding (Eq. 1) is continuous (1-Lipschitz), suppressing low-saliency noise by up to 20×20\times while letting strongly driven channels enter smoothly.

Thank you for the detailed response. I appreciate the authors' acknowledgment that the original Theorem 1 follows from the chosen parameterization. While the revisions are helpful, I remain unconvinced that they sufficiently address my concerns about the theoretical contribution and the overall clarity of the presentation. I therefore maintain my original score.

Thanks for your last suggestion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank Reviewers Emgs, 4bEg, and 2Y81 for their candid, detailed feedback. We take full responsibility for the dense presentation and table-assembly bugs in the initial submission. We have substantially revised the paper along four aspects:

  1. Concise, finding-focused Machine Learning presentation (Reviewers Emgs, 2Y81, 4bEg):

    • Rewrote the Title and Abstract without mathematical symbols, defined few-shot headroom in plain English at first mention, framed the Three-Fold Structured Inductive Bias (where to update, how to update, how to score updates; § 1–§ 3, Figure 1) in standard ML terms (per-channel output scaling, saliency-weighted ℓ1\ell_1 soft thresholding, leave-one-out rollout distillation, sign-consensus merging, validation rollback), and restructured § 4 Experiments (§ 4.1–§ 4.5) to pair two compact main-text anchor tables (Tables 2–3) with finding-focused prose and non-redundant multi-panel trend figures (Figures 2–4) while relocating granular task/scenario score tables to Appendix B–C.
    • Added concise end-to-end pseudocode (Algorithm 1, § 3, page 5), formatted Table 1 (§ 1, page 3) at native \footnotesize without \resizebox, moved the 228-run (4B–32B) phase-space regime analysis, threshold-free prompt-lift curve, qualitative case studies, and deployment takeaways (Figure 4, § 4.5, pages 9–10) into the main text so § 6 Conclusion finishes cleanly at the bottom of page 10 (followed on page 11 by § 7 Limitations, Ethics Statement, Reproducibility Statement, and an Appendix roadmap in Table 4), condensed the Appendix from 27 to 7 pages (Appendix A–C, Tables 5–17), and generated Figures 1–4 with Type 42 vector fonts (0 LaTeX warnings or overfull boxes).
  2. Fix manual copy-paste bugs and model names (Reviewer 2Y81 Ethics Flag & Q1; Reviewer 4bEg Q2): We used unsloth/gemma-* and google/gemma-* interchangeably across two machines, which caused the scripts to fall back to Llama-3-8B defaults. Every numerical inconsistency traced to the fallback or manual LaTeX copy-paste across splits/templates in the Appendix.

  3. Controlled PEFT baselines and expanded model evaluation (Reviewers 4bEg, 2Y81 Q2):

    • Evaluated LoRA (r=8r=8), (IA)^3, VeRA (r=256r=256), DoRA (r=8r=8), AlphaEdit-LoRA (r=8r=8), and unregularized All-Linear Gain + LOO (τ0=0\tau_0=0) on Meta-Llama-3-8B under the same leave-one-out rollout-distillation objective and rollback gate (§ 4.2, Table 2, Figure 3a, Appendix Tables 11, 13).
    • Evaluated google/gemma-4-E4B (4.5B, released in 2026) alongside expanding the full 38-task suite from 4 to 6 models (4B–32B, 228 full-suite runs; 7 models / 237 evaluations total): on Gemma-4-E4B, zero-shot accuracy improves from 41.70% to 48.19% Compiled →\to 48.32% Served (+6.62%, 31.4% headroom, p=3.13×10−2p = 3.13 \times 10^{-2}; § 4.4, Table 3, Appendix Tables 13, 16–17).
  4. Optimizer- and depth-aware theory replacing v1 Theorem 1 (Reviewers Emgs, 4bEg):

    • Moved single-layer kernel preservation to Proposition 1 (explicitly noted as a necessary rather than sufficient condition) and proved a two-part Theorem 1 (§ 2.2, Appendix § A.2–A.4): (a) SparseAdamW keeps inactive channels at an exact fixed point whereas dense AdamW induces scale-invariant Θ(η)\Theta(\eta) step noise, and (b) composing Lipschitz bounds across LL pre-norm RMSNorm, causal Softmax attention, and SwiGLU blocks bounds Layer-LL hidden-state drift by O(ηTmax⁡deff)\mathcal{O}(\eta T_{\max}\sqrt{d_{\text{eff}}}) and next-token KL divergence by O(η2Tmax⁡2deff)\mathcal{O}(\eta^2 T_{\max}^2 d_{\text{eff}}).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.