- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to compile a contextual skill into model weights, where the skill is present during calibration but removed at deployment, so its behavior must be absorbed into a low-rank weight update. The authors cast this as a shared-control problem over attention values and derive three structural limits—identifiability from finite calibration data, realizability under grouped-query parameter sharing, and low-rank representability—all measurable on the frozen model before training. Based on these, they propose Preflight, a training-free ordering of candidate compilations that ranks calibration sets, layers, and ranks before any adapter is trained. On a fully enumerated 90-configuration menu across 18 skill seeds, Preflight reaches 95% of the best configuration with 4.76× less candidate-training GPU time than random search, or 1.86× once its own cost is charged, and the ordering generalizes across skill families and model architectures.
Critical strengths
-
It reframes skill compilation as a pre-analyzable structural problem with three orthogonal, measurable limits. The paper derives identifiability (the causal-prefix bound in Proposition 1), realizability (the GQA condition in Eq. 4), and low-rank representability (Proposition 2), corresponding to the calibration set, the architecture, and the rank budget, all measurable on the frozen model before training. The validation in Fig. 2 is quantitative (Eq. 3 holds on 60/60 cells; Eq. 5 has a median observed/predicted ratio of 1.013). This turns design choices from hyperparameter search into decisions readable from structural signals.
-
The Preflight ordering is validated on a fully enumerated menu and reproduced prospectively. With 18 skill seeds, 90 configurations, and 1,656 training runs, the cost replay is exact rather than estimated. At the 95% target, Preflight reduces candidate-training time by 4.76× over random search, or 1.86× once its own cost is charged; the priority order was frozen to disk before any adapter trained and reproduced on a second skill family, Qwen3-8B, and Llama-3.1-8B (1.74×–5.82×), ruling out post-hoc selection.
-
Appendix S's "control geometry vs. task optimization" is a counterintuitive and independently valuable finding. A standard SFT recipe monotonically increases the control residual (from 1.0 to 3.56/5.78), with its minimum occurring at steps 5 and 8, before the task is learned. This shows that task optimization and the control object are anti-aligned, which directly informs why compilation sometimes works and sometimes does not.
Critical weaknesses
-
The paper is hard to read. The definitions of the core objects , , , are scattered across Section 2.2, Appendix E, and Appendix I, forcing the reader to jump back and forth to assemble the full picture. The notation is dense, and there are inconsistencies between the main text and the appendix (e.g., vs. ). How Stage II derives the predicted error in Eq. 5 from a single activation capture is not explained in the main text and requires Appendix Q. This directly affects both the reviewing experience and the assessment of the method's reproducibility.
-
The effectiveness of the ordering depends on the discriminative power of the menu, but no predictive criterion is provided. Table 6 shows that on Llama-3.1-8B the composite ordering falls behind both baselines ( vs. /), with one seed reversing. The authors attribute this to the low discriminative power of that menu (5–8 of 30 configurations reach the target), which implies that Preflight may be harmful on low-discriminative menus. Yet the paper gives no actionable criterion for when Preflight should be used.
-
The optimism of the Stage II predicted error is not quantified. Appendix Q explicitly states that the spectrum is computed per KV group and then summed, whereas the rank constraint applies to the combined update, making an optimistic lower bound. However, the actual gap is not reported. If the gap is large, Stage II may systematically favor higher ranks, undermining the reliability of the ordering.
Questions for authors
-
Can the authors provide an actionable criterion for when Preflight should be used? Table 6 shows that on Llama-3.1-8B the composite ordering falls behind both baselines ( vs. /), which the authors attribute to the low discriminative power of that menu. Is there a pre-training statistic that predicts menu discriminative power, so that a user can decide in advance whether Preflight will help or hurt?
-
Can the authors quantify the optimism of the Stage II predicted error? Appendix Q states that is an optimistic lower bound because the spectrum is computed per KV group and then summed, while the rank constraint applies to the combined update. What is the actual gap between this per-group estimate and the true combined rank- error on the experimental menu, and does this gap systematically bias Stage II toward higher ranks?
3 comments
Official Reply to Reviewer hVA4 (2/3)
W2: The effectiveness of the ordering depends on the discriminative power of the menu, but no predictive criterion is provided. Table 6 shows that on Llama-3.1-8B the composite ordering falls behind both baselines (1.74× vs. 2.88×/2.61×), with one seed reversing. The authors attribute this to the low discriminative power of that menu (5–8 of 30 configurations reach the target), which implies that Preflight may be harmful on low-discriminative menus. Yet the paper gives no actionable criterion for when Preflight should be used.
Q1: Can the authors provide an actionable criterion for when Preflight should be used? Table 6 shows that on Llama-3.1-8B the composite ordering falls behind both baselines (1.74× vs. 2.88×/2.61×), which the authors attribute to the low discriminative power of that menu. Is there a pre-training statistic that predicts menu discriminative power, so that a user can decide in advance whether Preflight will help or hurt?
Reply to W2 and Q1: Thank you for highlighting this important practical concern. We agree that an effective candidate ordering does not necessarily justify the cost of Preflight, particularly when the candidate menu has limited discriminative power. We add a practical adoption criterion and stopping protocol in §4.4, allowing users to decide whether to use Preflight before committing to a full search.
1. A Practical Criterion for Adopting Preflight.
Rather than relying on an indirect measure of menu discriminative power, we compare the expected search savings directly with the screening cost. Let denote screening overhead, the cost of training and validating one candidate, and and the numbers of candidates required to reach a validation target. Preflight is worthwhile when
Users can measure and on the target model and hardware, and estimate the number of candidates saved from similar tasks or a small pilot, whose cost is also included. This criterion accounts for menu discriminative power, model architecture, and hardware costs without requiring an exact prediction of the best configuration. We also specify a stopping protocol with a validation target and total budget fixed before searching.
2. Validation Across Models and Candidate Spaces.
The criterion agrees with the observed outcomes in all four tested settings:
| Model | Candidates | Screening cost | Candidates saved | Decision |
|---|---|---|---|---|
| Gemma-2-2B (main) | 90 | 3.6 | 6 | Use (1.86× net) |
| Gemma-2-2B (second) | 30 | 2.1 | 6 | Use (2.05× net) |
| Qwen3-8B (second) | 30 | 17.7 | 6 | Skip |
| Llama-3.1-8B (second) | 30 | 23.5 | 2 | Skip |
Screening cost is reported in candidate-training equivalents.
Our results provide a concrete decision rule: use Preflight only when the estimated number of candidates saved exceeds the screening cost measured in candidate-training equivalents; otherwise, skip it. This estimate can be obtained from comparable tasks or a small pilot before the full search. For Llama-3.1-8B, screening costs 23.5 candidate-training equivalents but saves only two candidates, so the rule recommends skipping Preflight even when the ordering improves. This makes the distinction clear: better ranking does not necessarily translate into net savings, and our criterion accounts for both.
Official Reply to Reviewer hVA4 (3/3)
W3: The optimism of the Stage II predicted error is not quantified. Appendix Q explicitly states that the spectrum is computed per KV group and then summed, whereas the rank constraint applies to the combined update, making an optimistic lower bound. However, the actual gap is not reported. If the gap is large, Stage II may systematically favor higher ranks, undermining the reliability of the ordering.
Q2: Can the authors quantify the optimism of the Stage II predicted error? Appendix Q states that is an optimistic lower bound because the spectrum is computed per KV group and then summed, while the rank constraint applies to the combined update. What is the actual gap between this per-group estimate and the true combined rank- error on the experimental menu, and does this gap systematically bias Stage II toward higher ranks?
Reply to W3 and Q2: Thank you for highlighting this important issue. We agree that the optimism of the group-wise relaxation needs to be quantified, particularly its effect on rank selection. We add a quantitative comparison in Appendix Q and clarify in §3 that Stage II uses the relaxed score for ranking rather than as an exact solution to the global rank constraint.
1. Quantifying the Relaxation Gap and Its Effect on Selection.
We compute two bounds on the optimal error under the shared global rank- constraint: the group-wise SVD solution gives a lower bound , while alternating least squares with multiple initializations provides a feasible upper bound .
The gap is small, and configuration selection remains stable. Across 20 seeds, the median relative gaps are 3.7%, 6.4%, and 4.0% at ranks 4, 16, and 64, respectively. The two rankings have a median Kendall's of 0.94, with 93% of configuration pairs having a provably fixed order based on non-overlapping bounds. Both methods retain identical candidate sets and select the same top configuration (L25, r64) in 20/20 seeds.
Importantly, the relaxation does not systematically favor higher ranks. The median error ratio between ranks 4 and 64 is 1.66 under the relaxation and 1.72 under the global constraint, indicating that the latter penalizes lower ranks more strongly.
2. Impact on Search Efficiency.
We further apply both rankings to the same trained candidates and compare their speedups over random search in reaching 95% of the best candidate performance.
| Rank constraint for scoring | Original 18 seeds | Prospective 6 seeds |
|---|---|---|
| Group-wise rank- (Preflight) | 4.76× | 4.03× |
| Global rank- (feasible ) | 4.71× | 4.35× |
The results are also consistent across additional task and model settings. The speedups remain unchanged on Gemma-2-2B (5.82×) and Qwen3-8B (4.87×), while global-rank scoring improves Llama-3.1-8B from 1.74× to 2.10×.
These results show that the group-wise relaxation slightly underestimates the global error, but preserves configuration selection and search efficiency without systematically favoring higher ranks. Its lower computational cost therefore makes it a suitable ranking proxy for Preflight.
We sincerely appreciate your constructive feedback, which has helped us improve the paper. Thank you again for your time and careful consideration. We welcome any further questions or discussion.
Official Reply to Reviewer hVA4 (1/3)
Reply to W1: Thank you for your careful reading. We apologize for the difficulty caused by the presentation of the theory in the main text. We make three revisions to improve clarity and reproducibility, with the modified sections marked in blue.
1. Clarifying the three execution settings. In §2.2, we now distinguish three executions when introducing matched-state routing (B0). Skill-conditioned execution defines the target behavior. The matched-state counterfactual fixes the retained states (ZS) and recomputes routing after removing the Skill, serving only to derive the local control relation. Skill-free execution represents the actual deployment setting, and is used for both Stage II scoring and all task evaluations of the compiled model.
2. Correcting inconsistencies between the main text and appendices. We correct Appendix E.5, where B0 is incorrectly called "compiled routing," to consistently refer to matched-state routing. We also replace the remaining "Stage III" references in Appendix U with "Stage II" and correct the reference to Eq. 3 in §3, which should instead point to the ranking key K.
3. Explaining the complete Stage II computation in the main text. We add a paragraph to §3 detailing how Eq. 5 is evaluated from activation captures. We select 64 probes of the most common sequence length from the calibration pool, shared across all candidates, and append the interpreter's answer to each. We run one forward pass with the Skill and one without it. For each KV group, we stack the value-projection inputs and head-output differences into Xg and Dg, then apply Eq. 5 to compute the error at each rank. Summing and normalizing across groups yields Eℓ,rprobe, which is combined with the 1.05 rank-dominance rule and Stage I labels to construct the final ranking key.
We believe these revisions make the theoretical framework easier to follow and the Preflight procedure easier to reproduce.