acceptodds
Under review as a conference paper at ICLR 2027

Beyond Semantic Relevance: Grounding Skill Selection in Model-Specific Execution Experience

Abstract

Reusable skills equip language-model agents with procedural knowledge for reasoning and problem solving. Access to a skill library, however, does not ensure that an agent can select the procedures it will benefit from. Skill descriptions express intended applicability, whereas their value depends on the executing model's capabilities and the question at hand. This mismatch creates a selection bottleneck for small and medium-sized models. We propose SkillCredit ValueFusion, which grounds semantic skill selection in the executor's own experience. Historical rollouts provide question-dependent estimates of skill gains. A lightweight value model then learns how the agent's current semantic preferences should correct those estimates, using within-question reward contrasts rather than a single best-skill label. Independent calibration adjusts the correction strength for each backbone and task domain. The resulting selector combines historical execution evidence with current judgment while keeping the language model and skill library fixed. Experiments across question-answering benchmarks and model backbones show significant overall improvements over agent self-selection and similarity-based retrieval. Further analyses expose complementary errors between semantic choice and historical credit, and show why their balance must be calibrated to the executor rather than imposed uniformly.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Agentic AI, Skill Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_64XkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Executor-Calibrated Skill Selection (ECSS), which selects a skill based on the executing model’s historical outcomes rather than semantic relevance alone. It combines a question-dependent credit predictor with a preference-informed reward correction and calibrates the correction strength for each executor–domain pair.

Critical strengths
  1. The distinction between semantic preference and actual execution benefit is meaningful. Treating “no skill” as an available action is also reasonable, since applying a seemingly relevant instruction can reduce answer quality.
  2. The paper includes useful controls, separate fitting and calibration partitions, and additional experiments examining calibration sensitivity and answer-generation variation.
Critical weaknesses
  1. The benefit of the complete method is not convincingly demonstrated. In Table 1, credit-only selection achieves an average F1 of 60.14, compared with 59.52 for full ECSS. The arithmetic extension similarly shows that historical credit accounts for nearly all the improvement on Mistral. Although these results support execution-grounded selection, they do not establish the benefit of the proposed preference-informed correction. The reduced calibration variation reported on MuSiQue comes with lower mean performance and does not extend to lower answer-generation variation. Given the additional fitting stage and semantic selection call, the necessity of the full design remains unclear.

  2. The methodological contribution beyond supervised utility prediction needs clearer justification. The credit predictor estimates observed skill gains using frozen embedding features and ridge regression, while the correction is another linear predictor fitted to residual reward differences. This is a reasonable implementation, but the paper should more clearly establish why this formulation is preferable to directly predicting action rewards.

  3. The necessity of question-dependent selection is insufficiently established. A useful control would select the best fixed skill using only fitting or calibration data and apply it to every evaluation question. Appendix H reports that credit-only selection chooses the same skill on all 256 target questions in one condition, suggesting that some gains may come from identifying an executor-specific default instruction rather than matching skills to individual questions. A systematic fixed-skill comparison would clarify this distinction.

  4. The practical benefit relative to data collection cost remains unclear. Credit fitting alone requires 8,192 answer generations per executor–domain pair, followed by additional executions for correction fitting and calibration. Full ECSS also requires an online preference call that credit-only selection avoids. The paper does not establish the quality–cost trade-off or how much receiving-model data is needed when another model’s history is already available. Since transferred or pooled credit sometimes outperforms own-executor credit, a comparison between independent fitting and low-budget adaptation would be particularly informative.

  5. The evaluation supports a relatively narrow setting. The experiments primarily use fixed, dataset-specific libraries and supplied-context QA, with arithmetic as an additional domain. This provides evidence for selecting among predefined reasoning instructions, but leaves open whether the approach remains effective when skills are added, modified, or used in interactive agent workflows.

Questions for authors

How does ECSS compare with the best fixed skill selected exclusively on fitting or calibration data?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 64Xk (Part 1/2)

Thank you for your constructive feedback on the methodological necessity of correction, question-dependent selection, and the practical scope of ECSS.

Weakness 1. The full ECSS selector provides additional quality gains on stronger backbones by complementing historical credit.

We agree that historical credit is a strong baseline and that correction does not consistently improve performance on the original smaller backbones. We hypothesize that this is partly due to their limited ability to exploit semantic information reliably. To investigate this, we conducted additional experiments on the more capable Qwen3.8-27B backbone.

Table 1. Qwen3.8-27B results (Token F1).

MethodMuSiQueHotpotQA2Wiki
No skill74.5389.6093.50
Best fixed skill74.6488.8093.99
Semantic-only72.7888.7294.21
ECSS (credit only)74.6488.8093.51
ECSS (full)74.9589.9394.39

ECSS (full) outperforms both standalone selectors and all other baselines across all three datasets, improving over credit-only by +0.31, +1.13, and +0.88 F1, respectively.

Historical credit provides an execution-grounded default, while correction learns when question-specific evidence warrants departing from it. The additional results suggest that such adjustments can improve quality when semantic judgments are sufficiently reliable.

The expanded five-stream evaluation does not establish lower generation variability for full ECSS, and credit-only remains stronger on the original smaller backbones. Thus, correction's value lies in its potential quality gains in suitable settings, rather than universally improving performance or generation stability.


Weakness 2. Residual correction benefits from an execution-grounded reference predictor rather than increased model expressivity.

The key distinction is the inductive bias: residual correction anchors predictions to observed executor-specific utility and learns question-specific deviations, whereas direct reward prediction estimates action rewards without this execution-grounded reference. With the same linear features, this changes the reference predictor and learning objective rather than increasing model expressivity.

To examine this design choice, we compared the two approaches under matched fitting and calibration settings.

Table 2. Residual correction versus direct reward prediction on Qwen3.8-27B (Token F1).

MethodHotpotQA2Wiki
Direct reward88.5894.21
ECSS (full)89.9394.39

ECSS (full) outperforms direct reward prediction on both datasets, with gains of +1.35 and +0.18 F1, respectively. Although direct reward performs slightly better on the original smaller-backbone average, these additional results suggest that an execution-grounded reference can benefit residual correction, particularly with stronger backbones. We do not claim that residual fitting is universally superior.


Weakness 3. Question-dependent selection can improve upon a strong fixed skill by adapting to individual questions.

We agree that identifying a strong executor-specific default skill can account for substantial gains. However, question-dependent selection allows ECSS to adapt this default to individual questions, including choosing no skill when additional guidance is unnecessary.

As shown in Table 1 (Weakness 1), ECSS (full) outperforms the best fixed skill across all three Qwen3.8-27B datasets, with gains of +0.31, +1.13, and +0.40 F1 on MuSiQue, HotpotQA, and 2Wiki, respectively.

To examine question-specific adaptation, we analyzed 128 MuSiQue questions. Compared with credit-only, ECSS (full) changes 65 selections, of which 5 improve F1, 4 worsen F1, and 56 leave F1 unchanged. All nine F1-changing cases involve switching between the default skill and no skill.

These observations illustrate how correction can depart from a strong default, although such departures are not always beneficial. Indeed, the best fixed skill outperforms ECSS on a separate MuSiQue evaluation (65.16 vs. 63.36 F1), indicating that question-dependent adaptation does not guarantee improvement.

AuthorsOpenReview ↗

Response to Reviewer 64Xk (Part 2/2)

Weakness 4. Donor-only transfer reduces receiver-side collection requirements, while additional history remains more cost-effective on smaller backbones.

We agree that executor-specific history collection introduces substantial upfront costs. Donor-only transfer provides a lower-collection-cost alternative by reusing existing source history and adapting through receiver calibration, without rebuilding receiver-specific execution history. We evaluated this trade-off across six calibration budgets.

Table 3. Receiver-calibration budget (mean Token F1 across eight conditions).

History source / CAL questions0163264128256
Own history59.0359.1258.8760.2460.3859.82
Pooled history57.5557.7658.1158.5858.3358.30
Donor-only history57.8557.5158.0057.8857.7158.41

Own history generally achieves higher F1, while donor-only transfer avoids collecting 8,192 receiver-specific historical answers, offering a practical alternative when source histories are already available. Calibration gains are nonmonotonic, so additional receiver data does not necessarily improve quality.

Table 4. Quality and collection-budget allocation.

MethodMean F1Charged tokens (M)
ECSS (credit only)61.0217.522
ECSS (credit only; more history)61.0721.787
ECSS (full)60.3524.497

Under the shared budget ceiling, additional credit history is the more effective allocation on the original smaller backbones. However, the Qwen3.8-27B results in Weakness 1 indicate that correction can provide additional quality benefits in other settings.

For online efficiency, ECSS (full) exhibits comparable end-to-end latency to credit-only on Qwen3.8-27B (22.50s vs. 23.33s), measured over 16 fixed questions and two serial passes. This does not establish a general speed advantage or isolate the additional latency of the semantic call.


Weakness 5. ECSS extends to additional reasoning tasks and evolving skill libraries, while interactive workflows remain unvalidated.

To evaluate generalization beyond fixed, benchmark-specific QA settings, we conducted additional experiments on BBH, covering diverse reasoning tasks, and evolving skill libraries with previously unseen skills.

Table 5. BBH results with Qwen3-4B (Exact Accuracy).

MethodAccuracy (%)
No skill71.67
Best fixed skill71.20
Credit correction71.85
ECSS (full)72.87

540 questions across 27 configurations, averaged over two answer streams.

ECSS (full) achieves the highest accuracy on BBH, outperforming no skill by +1.20 percentage points and credit correction by +1.02 points. This provides additional evidence of applicability beyond multi-hop QA.

We further evaluated whether ECSS can incorporate previously unseen skills without refitting.

Table 6. Generalization to unseen skills (Qwen3.8-27B, MuSiQue).

MethodFold 1Fold 2Fold 3Fold 4Avg.
ECSS (full; seen skills only)72.9474.1372.8174.9573.71
ECSS (full; all skills)74.6575.5973.4274.8374.62
Unseen skill selections69/12847/1286/12830/128152/512

Introducing unseen skills improves mean F1 from 73.71 to 74.62, while previously unseen skills are selected in 152 of 512 evaluations. The selection frequency confirms that new skills participate in decisions, while the F1 improvement provides separate evidence of their potential benefit.

This capability is enabled by shared predictors that learn transferable selection patterns, semantic features that represent new skill descriptions, and a neutral mean-credit prior for skills without execution history. Together, these components allow ECSS to score and select unseen skills without refitting.

These results support generalization to additional reasoning tasks and evolving skill libraries within an existing domain, but not yet to interactive agent workflows. Such workflows require estimating how skill choices affect subsequent states and long-horizon outcomes, which remains an important direction for future work.


Question 1. The best fixed skill is selected without evaluation data, and full ECSS outperforms it in several additional settings.

The best fixed skill is selected exclusively using historical fitting outcomes for QA and 135 calibration questions for BBH, without accessing evaluation data. The selected skill is then applied unchanged to every evaluation question.

As shown in Table 1 (Weakness 1), ECSS (full) outperforms the best fixed skill on all three Qwen3.8-27B datasets, with gains of +0.31, +1.13, and +0.40 F1 on MuSiQue, HotpotQA, and 2Wiki, respectively. Similarly, Table 5 (Weakness 5) shows higher accuracy on BBH (72.87% vs. 71.20%).

Thank you for your response; my concerns have been addressed, so I have raised my score to 3.

AuthorsOpenReview ↗

Response to Reviewer 64Xk

Thank you for your reconsideration and for updating score. We sincerely appreciate your time and constructive feedback.

Reviewer_4rxVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies skill selection for a fixed language-model executor. It proposes Executor-Calibrated Skill Selection (ECSS), which estimates the benefit of each skill from the executor's own execution history, then optionally adds a preference-informed residual correction whose strength is selected on held-out calibration questions. Across MuSiQue, HotpotQA, 2Wiki, and GSM8K, using Qwen3 and Mistral backbones, the execution-grounded selectors generally outperform direct agent self-selection and retrieval baselines. The main QA result is 59.52 Token F1 for ECSS(full) versus 57.04 for Agent; the simpler credit-only variant is higher at 60.14. The evaluation uses disjoint fitting, correction, calibration, and evaluation question sets, paired comparisons, bootstrap intervals, component controls, and repeatability studies.

Critical strengths

1.A key strength is the paper’s observation that the usefulness of a skill depends on the specific model that executes it, rather than being an intrinsic property of the skill description. This is well motivated by the reported failures of direct agent self-selection and by the cross-backbone transfer results, which show that skill credit learned from one model may not transfer reliably to another. This executor-specific perspective is conceptually useful and provides a clear motivation for the proposed method. 2.The experimental evaluation is relatively broad and includes several useful controls. The paper evaluates ECSS across multiple multi-hop QA datasets and several model backbones, and further extends the analysis to GSM8K. In addition to reporting aggregate performance, the paper includes ablations on calibration, semantic correction, transfer, execution-history size, and repeatability. These analyses make it easier to understand when the method helps and which components are responsible for the gains, which strengthens the empirical credibility of the work. 3.The method is designed with a relatively clear separation between fitting, calibration, and evaluation.ECSS uses disjoint data splits for credit estimation, correction fitting, calibration, and final evaluation, which reduces the risk of directly tuning the selector on the evaluation set. The paper also reports paired bootstrap confidence intervals for the main comparison. This experimental discipline is a positive aspect of the paper and makes the reported gains more interpretable.

Critical weaknesses

1.The added value of the full ECSS formulation over the simpler credit-only variant is not sufficiently convincing.In the main results, the credit-only version achieves a higher average F1 than full ECSS (60.14 vs. 59.52), and the full method does not consistently outperform the simpler variant across conditions. This weakens the case for the preference-informed correction, which is one of the main methodological additions beyond historical credit. Since the extra component introduces additional modeling and calibration complexity, I would like to see clearer evidence that it provides a consistent benefit rather than only helping in selected settings. 2.The empirical gains are somewhat uneven across tasks and model backbones, which raises questions about robustness and generalization.Full ECSS outperforms direct Agent selection in only five of the eight primary conditions, and the ablation results show that semantic preference features can help in some settings but hurt in others. The paper also reports that the QuALITY extension did not reproduce the main gains. These results suggest that the method’s effectiveness is still fairly dependent on the particular executor and task domain, limiting confidence in its broader applicability. 3.The offline data collection cost is substantial, and the practical efficiency trade-off is not fully evaluated.The method requires executor-specific execution histories for each executor–domain pair, with thousands of answer generations used for credit estimation, correction fitting, and calibration. The paper acknowledges that end-to-end latency and equal-budget deployment utility have not been measured. This is important because the practical advantage of ECSS depends not only on answer quality, but also on whether the performance gains justify the additional offline collection and calibration cost.

Questions for authors

1.Is full ECSS considered as the final recommended method, or should the core contribution of the paper be more understood as executor-specific credit estimation, with preference correction being merely an optional module? 2.How sensitive are the reported gains to the amount and cost of executor-specific execution history?The method requires substantial offline outcome collection for each executor–domain pair. Could the authors provide a clearer comparison of performance under matched data-collection or computation budgets, especially against simpler baselines such as credit-only selection? This would help assess whether the observed gains justify the additional complexity and cost. 3.What evidence supports generalization beyond the current fixed-skill, benchmark-specific setting?The current evaluation uses fixed skill libraries and mostly benchmark-specific fitting, and the QuALITY extension does not reproduce the main gains. Could the authors clarify which parts of ECSS they expect to generalize to new domains, evolving skill libraries, or unseen task distributions without collecting a new executor-specific history? A concrete answer here would significantly affect my confidence in the broader applicability of the method; broader validation itself could be considered future work.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4rxV (Part 1/2)

Thank you for your constructive feedback on the conditions under which ECSS is effective, its practical efficiency, and its broader applicability.

Weakness 1. Semantic correction complements historical credit, with benefits that depend on backbone capabilities.

We agree that historical credit provides a strong baseline and that correction does not consistently improve performance on the original smaller backbones. We hypothesize that this is partly due to their limited ability to exploit semantic information reliably. To investigate this, we conducted additional experiments on the more capable Qwen3.8-27B backbone.

Table 1. Qwen3.8-27B results (Token F1).

MethodMuSiQueHotpotQA2Wiki
No skill74.5389.6093.50
Best fixed skill74.6488.8093.99
Semantic-only72.7888.7294.21
ECSS (credit only)74.6488.8093.51
ECSS (full)74.9589.9394.39

ECSS (full) outperforms both standalone selectors and all other baselines across all three datasets, improving over credit-only by +0.31, +1.13, and +0.88 F1, respectively.

Historical credit provides an execution-grounded default, while correction learns when question-specific evidence warrants departing from that default. These results suggest that semantic correction can provide complementary value when supported by sufficiently reliable semantic judgments. We do not claim that larger models necessarily select skills more accurately; rather, the observed benefits appear to depend on the backbone.


Weakness 2. Additional BBH experiments demonstrate benefits beyond multi-hop QA, although performance remains task-dependent.

We agree that skill effectiveness depends on both task characteristics and executor capabilities. A skill may provide a missing reasoning operation but introduce unnecessary complexity when the executor can already solve the question directly. To evaluate ECSS beyond multi-hop QA and mathematical reasoning, we conducted additional experiments on BBH, covering logical reasoning, symbolic manipulation, state tracking, and language understanding.

Table 2. BBH results with Qwen3-4B (Exact Accuracy).

MethodAccuracy (%)
No skill71.67
Best fixed skill71.20
Credit correction71.85
ECSS (full)72.87

Best fixed skill is selected from 25 skills using 135 calibration questions.

ECSS (full) achieves the highest accuracy, outperforming no skill by +1.20 percentage points and credit correction by +1.02 points. This provides additional evidence that ECSS can benefit reasoning tasks beyond multi-hop QA. While QuALITY remains a negative result, the additional BBH experiments provide evidence of ECSS's applicability to a broader range of reasoning tasks.


Weakness 3. Additional history is more cost-effective on smaller backbones, while correction introduces comparable online latency.

We conducted additional experiments to assess whether the correction budget would be better allocated to collecting more execution history.

Table 3. Quality and collection-budget allocation.

MethodMean F1Charged tokens (M)
ECSS (credit only)61.0217.522
ECSS (credit only; more history)61.0721.787
ECSS (full)60.3524.497

Collecting more execution history remains the more effective budget allocation on the original smaller backbones. However, as shown in Weakness 1, correction provides additional quality gains on Qwen3.8-27B, suggesting that its benefits depend on backbone capability.

Table 4. Serial end-to-end latency on Qwen3.8-27B.

MethodMean (s)P90 (s)
ECSS (credit only)23.3344.31
ECSS (full)22.5042.84

ECSS (full) exhibits comparable end-to-end latency to credit-only in the measured workload (22.50s vs. 23.33s). This does not establish a general speed advantage or isolate the semantic call's incremental latency, since instruction choice and answer length also affect total inference time.

AuthorsOpenReview ↗

Response to Reviewer 4rxV (Part 2/2)

Question 1. Credit-only is preferable when efficiency dominates, while full ECSS offers additional value when semantic correction is effective.

We agree that credit-only is a strong and more cost-effective choice on the original smaller backbones. We recommend credit-only when collection efficiency is the primary concern, while retaining full ECSS in settings where correction provides measurable quality benefits. As shown in Table 1 (Weakness 1), full ECSS outperforms credit-only across all three Qwen3.8-27B datasets.

Methodologically, historical credit estimates executor-specific skill utility, while correction enables question-specific deviations from these historical preferences, with calibration controlling the influence of those adjustments. Thus, correction is a complementary mechanism rather than a universally beneficial replacement for historical credit.


Question 2. More execution history improves performance, whereas receiver-calibration gains are nonmonotonic.

We conducted additional experiments to evaluate sensitivity to execution-history size and receiver-calibration budget, keeping evaluation questions fixed.

Table 5. History-size sensitivity (mean Token F1 across eight conditions).

History questionsHistorical answersMean F1
1281,02457.93
2562,04858.17
5124,09659.03
1,0248,19259.82

Increasing history from 128 to 1,024 questions improves mean F1 from 57.93 to 59.82, suggesting that additional execution evidence helps capture recurring executor-specific behavior.

Table 6. Receiver-calibration budget (mean Token F1 across eight conditions).

History source / CAL questions0163264128256
Own history59.0359.1258.8760.2460.3859.82
Pooled history57.5557.7658.1158.5858.3358.30
Donor-only history57.8557.5158.0057.8857.7158.41

History collection and receiver calibration play distinct roles: additional history improves coverage of observed executor behavior, whereas receiver-calibration gains are nonmonotonic. Own history reaches its highest observed F1 at 128 calibration questions, but we do not select an intermediate budget based on evaluation performance.

As shown in Table 3 (Weakness 3), allocating additional tokens to history collection is more effective than correction on the original smaller backbones under the shared budget ceiling.


Question 3. ECSS can incorporate previously unseen skills without refitting, while cross-domain generalization remains an open question.

To evaluate generalization to evolving skill libraries, we conducted additional experiments in which approximately 25% of skills were held out during fitting and introduced only at evaluation, without refitting.

Table 7. Generalization to unseen skills (Qwen3.8-27B, MuSiQue).

MethodFold 1Fold 2Fold 3Fold 4Avg.
ECSS (full; seen skills only)72.9474.1372.8174.9573.71
ECSS (full; all skills)74.6575.5973.4274.8374.62
Unseen skill selections69/12847/1286/12830/128152/512

Fitting uses 6–7 of 9 skills; evaluation introduces held-out skills without refitting. The first two rows report Token F1; the last row reports selection counts.

Introducing unseen skills improves mean F1 from 73.71 to 74.62, while the selector chooses previously unseen skills in 152 of 512 evaluations. The selection counts establish that the new skills are used, while the F1 improvement provides separate evidence that library expansion can be beneficial.

This capability is enabled by three design choices: (1) shared predictors learn transferable selection patterns rather than skill-specific parameters; (2) semantic features represent new skill descriptions and their relevance to the current question; and (3) a neutral mean-credit prior allows skills without execution history to participate in selection without assigning unsupported utility estimates.

Together, these components allow ECSS to score and select previously unseen skills without refitting or collecting execution history for the newly introduced skills.

While one fold declines slightly, these results support adaptation to evolving skill libraries within an existing domain, not zero-shot generalization to entirely new domains. Distribution shifts that alter executor-specific skill utility may still require new execution history.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer jgWD (Part 1/2)

Thank you for your careful assessment and constructive questions regarding the incremental value of correction, collection costs, and executor-specific history.

Weakness 1. Semantic correction provides complementary benefits on stronger backbones, while historical credit remains more cost-effective on smaller models.

We agree that historical credit is a strong baseline and that correction does not consistently improve performance on the original smaller backbones. We hypothesize that this is partly due to their limited ability to exploit semantic information reliably. To investigate this, we conducted additional experiments on the more capable Qwen3.8-27B backbone.

Table 1. Qwen3.8-27B results (Token F1).

MethodMuSiQueHotpotQA2Wiki
No skill74.5389.6093.50
Best fixed skill74.6488.8093.99
Semantic-only72.7888.7294.21
ECSS (credit only)74.6488.8093.51
ECSS (full)74.9589.9394.39

ECSS (full) outperforms both standalone selectors and all other baselines across all three datasets, improving over credit-only by +0.31, +1.13, and +0.88 F1, respectively. Historical credit provides an execution-grounded default, while correction learns when question-specific evidence warrants departing from it. These results suggest that correction can provide complementary value when semantic judgments are sufficiently informative, rather than universally improving performance across backbones.

Our expanded five-stream evaluation confirms the original credit-only advantage but does not establish lower generation variability for ECSS (full). Thus, correction's motivation lies in its potential quality gains, not improved generation stability.

Table 2. Quality and collection-budget allocation.

MethodMean F1Charged tokens (M)
ECSS (credit only)61.0217.522
ECSS (credit only; more history)61.0721.787
ECSS (full)60.3524.497

Expanded-history and full share the same per-condition token budget ceiling, with different actual token consumption.

Under this budget ceiling, additional credit history remains the more effective allocation on the original smaller backbones. We therefore position correction as a complementary mechanism whose benefits depend on the reliability of question-specific adjustments, rather than a universally preferable replacement for historical credit.


Weakness 2. Executor-specific history retains its advantage on new questions, despite the additional collection cost.

We agree that the original evidence was limited to the development panels. To address this, we conducted additional experiments with 256 new calibration and 256 new evaluation questions per condition.

Table 3. Receiver-calibrated history transfer on new questions (Token F1).

History sourceMu/4BMu/14BMu/M7BHot/4BHot/14BHot/M7B2Wiki/4B2Wiki/14BAvg.
Own history56.9355.9930.4079.8375.3636.6081.6161.8759.82
Pooled history59.6056.2426.0976.7272.1734.3279.3061.9858.30
Donor-only history61.9454.4526.3276.2272.1333.2480.8062.1658.41

Mu: MuSiQue; Hot: HotpotQA; 4B/14B: Qwen3; M7B: Mistral-7B.

Own history achieves the highest average F1 (59.82), outperforming pooled (58.30) and donor-only history (58.41), and wins 5 of 8 conditions. These results suggest that executor alignment can matter more than additional cross-model observations, although transfer remains beneficial in some conditions.

Regarding collection cost, pooled history includes receiver outcomes, whereas donor-only history excludes them. We distinguish source-history acquisition from receiver-side adaptation and acknowledge the additional upfront cost of collecting executor-specific histories.

AuthorsOpenReview ↗

Response to Reviewer jgWD (Part 2/2)

Question 1. Matched controls suggest that semantic information contributes beyond additional correction fitting, although the gains are not uniform.

Yes. We conducted matched experiments across all eight primary conditions, using identical candidate support, reward labels, fitting/calibration splits, and parameter searches. Credit correction uses only historical credit features, while shuffled semantics disrupts semantic alignment while retaining the correction architecture.

Table 4. Matched correction controls across eight primary conditions (Token F1).

MethodMu/4BMu/14BMu/M7BHot/4BHot/14BHot/M7B2Wiki/4B2Wiki/14BAvg.
Credit correction51.1149.2226.7783.1175.3644.2584.2367.1360.15
Shuffled semantics55.8549.2227.1284.1875.3636.4384.6266.4759.91
ECSS (credit only)56.1850.0426.7784.6075.3644.4684.6066.2061.02
ECSS (full)50.9049.6827.2683.2374.3544.2184.6268.5860.35

ECSS (full) modestly exceeds credit correction (+0.20) and shuffled semantics (+0.44) on average, consistent with a contribution from aligned semantic information beyond additional fitting alone. However, this effect is not uniform, and credit-only remains stronger overall on the original smaller backbones.

The Qwen3.8-27B results in Table 1 provide complementary evidence: full ECSS outperforms both standalone selectors across all three datasets. Together, these findings support the potential value of semantic correction while indicating that its effectiveness depends on the backbone.


Question 2. Own-model history generalizes to new questions, while independent recollection variability remains unquantified.

As shown in Table 3 (Weakness 2), own history retains the highest mean F1 (59.82) on the new-question confirmation cohort, outperforming pooled (58.30) and donor-only history (58.41) after receiver-side calibration. It performs best in 5 of 8 conditions, providing further evidence for executor-specific utility estimation beyond the development panels.

Independent full history recollection and refitting have not been repeated, so we cannot yet quantify this source of variability. The five-stream evaluation measures answer-generation variation with fixed histories and selection decisions, rather than variability from independently recollecting and refitting execution histories.

AuthorsOpenReview ↗

Response to Reviewer jgWD

We sincerely thank the reviewer for their thoughtful comments. We would greatly appreciate it if you could let us know whether our rebuttal has adequately addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.