- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Executor-Calibrated Skill Selection (ECSS), which selects a skill based on the executing model’s historical outcomes rather than semantic relevance alone. It combines a question-dependent credit predictor with a preference-informed reward correction and calibrates the correction strength for each executor–domain pair.
Critical strengths
- The distinction between semantic preference and actual execution benefit is meaningful. Treating “no skill” as an available action is also reasonable, since applying a seemingly relevant instruction can reduce answer quality.
- The paper includes useful controls, separate fitting and calibration partitions, and additional experiments examining calibration sensitivity and answer-generation variation.
Critical weaknesses
-
The benefit of the complete method is not convincingly demonstrated. In Table 1, credit-only selection achieves an average F1 of 60.14, compared with 59.52 for full ECSS. The arithmetic extension similarly shows that historical credit accounts for nearly all the improvement on Mistral. Although these results support execution-grounded selection, they do not establish the benefit of the proposed preference-informed correction. The reduced calibration variation reported on MuSiQue comes with lower mean performance and does not extend to lower answer-generation variation. Given the additional fitting stage and semantic selection call, the necessity of the full design remains unclear.
-
The methodological contribution beyond supervised utility prediction needs clearer justification. The credit predictor estimates observed skill gains using frozen embedding features and ridge regression, while the correction is another linear predictor fitted to residual reward differences. This is a reasonable implementation, but the paper should more clearly establish why this formulation is preferable to directly predicting action rewards.
-
The necessity of question-dependent selection is insufficiently established. A useful control would select the best fixed skill using only fitting or calibration data and apply it to every evaluation question. Appendix H reports that credit-only selection chooses the same skill on all 256 target questions in one condition, suggesting that some gains may come from identifying an executor-specific default instruction rather than matching skills to individual questions. A systematic fixed-skill comparison would clarify this distinction.
-
The practical benefit relative to data collection cost remains unclear. Credit fitting alone requires 8,192 answer generations per executor–domain pair, followed by additional executions for correction fitting and calibration. Full ECSS also requires an online preference call that credit-only selection avoids. The paper does not establish the quality–cost trade-off or how much receiving-model data is needed when another model’s history is already available. Since transferred or pooled credit sometimes outperforms own-executor credit, a comparison between independent fitting and low-budget adaptation would be particularly informative.
-
The evaluation supports a relatively narrow setting. The experiments primarily use fixed, dataset-specific libraries and supplied-context QA, with arithmetic as an additional domain. This provides evidence for selecting among predefined reasoning instructions, but leaves open whether the approach remains effective when skills are added, modified, or used in interactive agent workflows.
Questions for authors
How does ECSS compare with the best fixed skill selected exclusively on fitting or calibration data?
4 comments
Response to Reviewer 64Xk (Part 2/2)
Weakness 4. Donor-only transfer reduces receiver-side collection requirements, while additional history remains more cost-effective on smaller backbones.
We agree that executor-specific history collection introduces substantial upfront costs. Donor-only transfer provides a lower-collection-cost alternative by reusing existing source history and adapting through receiver calibration, without rebuilding receiver-specific execution history. We evaluated this trade-off across six calibration budgets.
Table 3. Receiver-calibration budget (mean Token F1 across eight conditions).
| History source / CAL questions | 0 | 16 | 32 | 64 | 128 | 256 |
|---|---|---|---|---|---|---|
| Own history | 59.03 | 59.12 | 58.87 | 60.24 | 60.38 | 59.82 |
| Pooled history | 57.55 | 57.76 | 58.11 | 58.58 | 58.33 | 58.30 |
| Donor-only history | 57.85 | 57.51 | 58.00 | 57.88 | 57.71 | 58.41 |
Own history generally achieves higher F1, while donor-only transfer avoids collecting 8,192 receiver-specific historical answers, offering a practical alternative when source histories are already available. Calibration gains are nonmonotonic, so additional receiver data does not necessarily improve quality.
Table 4. Quality and collection-budget allocation.
| Method | Mean F1 | Charged tokens (M) |
|---|---|---|
| ECSS (credit only) | 61.02 | 17.522 |
| ECSS (credit only; more history) | 61.07 | 21.787 |
| ECSS (full) | 60.35 | 24.497 |
Under the shared budget ceiling, additional credit history is the more effective allocation on the original smaller backbones. However, the Qwen3.8-27B results in Weakness 1 indicate that correction can provide additional quality benefits in other settings.
For online efficiency, ECSS (full) exhibits comparable end-to-end latency to credit-only on Qwen3.8-27B (22.50s vs. 23.33s), measured over 16 fixed questions and two serial passes. This does not establish a general speed advantage or isolate the additional latency of the semantic call.
Weakness 5. ECSS extends to additional reasoning tasks and evolving skill libraries, while interactive workflows remain unvalidated.
To evaluate generalization beyond fixed, benchmark-specific QA settings, we conducted additional experiments on BBH, covering diverse reasoning tasks, and evolving skill libraries with previously unseen skills.
Table 5. BBH results with Qwen3-4B (Exact Accuracy).
| Method | Accuracy (%) |
|---|---|
| No skill | 71.67 |
| Best fixed skill | 71.20 |
| Credit correction | 71.85 |
| ECSS (full) | 72.87 |
540 questions across 27 configurations, averaged over two answer streams.
ECSS (full) achieves the highest accuracy on BBH, outperforming no skill by +1.20 percentage points and credit correction by +1.02 points. This provides additional evidence of applicability beyond multi-hop QA.
We further evaluated whether ECSS can incorporate previously unseen skills without refitting.
Table 6. Generalization to unseen skills (Qwen3.8-27B, MuSiQue).
| Method | Fold 1 | Fold 2 | Fold 3 | Fold 4 | Avg. |
|---|---|---|---|---|---|
| ECSS (full; seen skills only) | 72.94 | 74.13 | 72.81 | 74.95 | 73.71 |
| ECSS (full; all skills) | 74.65 | 75.59 | 73.42 | 74.83 | 74.62 |
| Unseen skill selections | 69/128 | 47/128 | 6/128 | 30/128 | 152/512 |
Introducing unseen skills improves mean F1 from 73.71 to 74.62, while previously unseen skills are selected in 152 of 512 evaluations. The selection frequency confirms that new skills participate in decisions, while the F1 improvement provides separate evidence of their potential benefit.
This capability is enabled by shared predictors that learn transferable selection patterns, semantic features that represent new skill descriptions, and a neutral mean-credit prior for skills without execution history. Together, these components allow ECSS to score and select unseen skills without refitting.
These results support generalization to additional reasoning tasks and evolving skill libraries within an existing domain, but not yet to interactive agent workflows. Such workflows require estimating how skill choices affect subsequent states and long-horizon outcomes, which remains an important direction for future work.
Question 1. The best fixed skill is selected without evaluation data, and full ECSS outperforms it in several additional settings.
The best fixed skill is selected exclusively using historical fitting outcomes for QA and 135 calibration questions for BBH, without accessing evaluation data. The selected skill is then applied unchanged to every evaluation question.
As shown in Table 1 (Weakness 1), ECSS (full) outperforms the best fixed skill on all three Qwen3.8-27B datasets, with gains of +0.31, +1.13, and +0.40 F1 on MuSiQue, HotpotQA, and 2Wiki, respectively. Similarly, Table 5 (Weakness 5) shows higher accuracy on BBH (72.87% vs. 71.20%).
Thank you for your response; my concerns have been addressed, so I have raised my score to 3.
Response to Reviewer 64Xk
Thank you for your reconsideration and for updating score. We sincerely appreciate your time and constructive feedback.
Response to Reviewer 64Xk (Part 1/2)
Thank you for your constructive feedback on the methodological necessity of correction, question-dependent selection, and the practical scope of ECSS.
Weakness 1. The full ECSS selector provides additional quality gains on stronger backbones by complementing historical credit.
We agree that historical credit is a strong baseline and that correction does not consistently improve performance on the original smaller backbones. We hypothesize that this is partly due to their limited ability to exploit semantic information reliably. To investigate this, we conducted additional experiments on the more capable Qwen3.8-27B backbone.
Table 1. Qwen3.8-27B results (Token F1).
ECSS (full) outperforms both standalone selectors and all other baselines across all three datasets, improving over credit-only by +0.31, +1.13, and +0.88 F1, respectively.
Historical credit provides an execution-grounded default, while correction learns when question-specific evidence warrants departing from it. The additional results suggest that such adjustments can improve quality when semantic judgments are sufficiently reliable.
The expanded five-stream evaluation does not establish lower generation variability for full ECSS, and credit-only remains stronger on the original smaller backbones. Thus, correction's value lies in its potential quality gains in suitable settings, rather than universally improving performance or generation stability.
Weakness 2. Residual correction benefits from an execution-grounded reference predictor rather than increased model expressivity.
The key distinction is the inductive bias: residual correction anchors predictions to observed executor-specific utility and learns question-specific deviations, whereas direct reward prediction estimates action rewards without this execution-grounded reference. With the same linear features, this changes the reference predictor and learning objective rather than increasing model expressivity.
To examine this design choice, we compared the two approaches under matched fitting and calibration settings.
Table 2. Residual correction versus direct reward prediction on Qwen3.8-27B (Token F1).
ECSS (full) outperforms direct reward prediction on both datasets, with gains of +1.35 and +0.18 F1, respectively. Although direct reward performs slightly better on the original smaller-backbone average, these additional results suggest that an execution-grounded reference can benefit residual correction, particularly with stronger backbones. We do not claim that residual fitting is universally superior.
Weakness 3. Question-dependent selection can improve upon a strong fixed skill by adapting to individual questions.
We agree that identifying a strong executor-specific default skill can account for substantial gains. However, question-dependent selection allows ECSS to adapt this default to individual questions, including choosing no skill when additional guidance is unnecessary.
As shown in Table 1 (Weakness 1), ECSS (full) outperforms the best fixed skill across all three Qwen3.8-27B datasets, with gains of +0.31, +1.13, and +0.40 F1 on MuSiQue, HotpotQA, and 2Wiki, respectively.
To examine question-specific adaptation, we analyzed 128 MuSiQue questions. Compared with credit-only, ECSS (full) changes 65 selections, of which 5 improve F1, 4 worsen F1, and 56 leave F1 unchanged. All nine F1-changing cases involve switching between the default skill and no skill.
These observations illustrate how correction can depart from a strong default, although such departures are not always beneficial. Indeed, the best fixed skill outperforms ECSS on a separate MuSiQue evaluation (65.16 vs. 63.36 F1), indicating that question-dependent adaptation does not guarantee improvement.