acceptodds
Withdrawn from ICLR 2027

Group-Relative Skill Optimization for Frozen Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Optimizing external natural-language skills has emerged as an effective approach to adapting frozen large language models. Existing methods have improved task performance and stabilized skill updates through held-out validation and incremental editing. However, we identify two limitations of batch-level skill optimization: (1) common-denominator bias, where reflection over trajectories from different questions favors generic guidance over question-specific knowledge; and (2) all-or-nothing acceptance, where edits from different questions are accepted or rejected as a single update, making it difficult to retain useful edits while rejecting harmful ones. We introduce Group-Relative Skill Optimization (GRSO), which aligns reflection and verification at the question level. For each question, GRSO samples a group of trajectories and uses within-question reflection to propose itemized edits, contrasting successful and failed attempts. Each edit bundle is verified on its source question before accepted bundles are merged. Dynamic sampling and retry concentrate additional computation on difficult questions. This design preserves question-specific learning signals without held-out validation during optimization. Across SearchQA, DocVQA, and LiveMathBench, GRSO achieves the best or tied-best accuracy in all six target–benchmark settings. Compared with the unadapted models, it improves average accuracy across the three benchmarks by 15.2 and 22.0 percentage points for Qwen3.5-4B and Seed1.8, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Skill Optimization, Frozen Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7Uq6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Group-Relative Skill Optimization (GRSO), a framework for improving frozen language models by optimizing external natural-language skills. It aims to address two limitations of existing skill optimization methods: reflection over different questions can produce overly generic guidance, and evaluating an entire batch of edits together can reject useful edits alongside harmful ones. GRSO generates multiple trajectories for each training question and uses within-question reflection, contrasting successful and failed attempts when available, to propose itemized skill edits. Each question’s edit bundle is verified on its source question before accepted bundles are consolidated into a shared skill. Additional sampling and edit retries allocate more computation to difficult questions. The framework keeps model parameters fixed and does not require held-out validation during optimization. Experiments on retrieval QA, document understanding, and mathematical multiple-choice QA, using Qwen3.5-4B and Seed1.8, report best or tied-best accuracy across the six evaluated settings, with the largest improvements on the math benchmark.

Critical strengths
  1. The paper presents a clear and practical framework for optimizing external skills for frozen language models. The use of within-question trajectory comparisons and question-level edit verification is intuitive and enables selective retention of proposed updates.
  2. The approach avoids repeated full validation set checks, with reported reductions in training-token consumption.
Critical weaknesses
  1. The paper seems to use a wrong reference of the LiveMathBench benchmark. Based on the paper's description, it should be the LiveMathematicianBench following the SkillOpt paper, which evaluates multiple-choice questions.
  2. The paper shows the most significant improvements on the LiveMathBench benchmark compared to prior methods. However, based on Figure 2, the presented skills learned with GSPO include "Select the meta-option asserting a stronger result can be proven.”, which is a clear shortcut to this benchmark. The underlying benchmark, LiveMathematicianBench [1], constructs some questions by replacing the correct option with a distinctive statement that another option is correct but a stronger result can be proved. So the current performance on this benchmark could be using the shortcut rather than genuine generalization capability. The paper should redo the eval by removing this type of skill.
  3. Given one of the main limitations the paper identified is common-denominator bias, I would expect a more direct comparison of within-question reflection against mixed-question reflection while holding the remaining procedure fixed. The comparison with SkillOpt still mixed many other confoundings.
  4. The retained-update rates are difficult to interpret comparatively because GRSO and SkillOpt use different acceptance criteria and numbers of candidate bundles per step. A higher retention rate does not itself establish more useful learning. Accuracy-versus-compute curves would better substantiate the efficiency claim, complementing the costs already reported in Table 3.
  5. More implementation details should be provided for reproduction.

[1] He, Linyang, et al. "LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketches." Third Conference on Language Modeling.

Questions for authors

See weaknesses.

Reviewer_d5twOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposed GSRO, optimizing natural-language skills for a frozen LLM. The authors tackle two problems with previous batch-level optimization methods: common-denominator bias and all-or-nothing acceptance due to coarse validation. To resolve this, GSRO samples a group of trajectories per question under the current skill. An optimizer contrasts the success and the failure to propose an itemized edit bundle. Accepted bundles from the batch are then merged into the skill. GSRO achieves best or tied-best in all six benchmark settings.

Critical strengths
  • Clear motivation. The two failure modes are well motivated, and the framing, unit of verification vs. unit of learning, is intuitive.
  • Strong results. GSRO achieves consistent wins and large gains at LiveMathBench compared to SkillOpt.
  • Efficient method. GSRO requires fewer rollouts compared to baselines (but not at LiveMathBench), and most steps produce an update.
Critical weaknesses
  • Limited component novelty. Each components has been proposed in the previous work and this work is close to combination of prevoius work.
    • Within-question self-contrast over multiple trajectories is close to ReasoningBank (ICLR’26).
    • Itemized incremental edits are close to ACE (ICLR’26).
    • The local acceptance check is close to GEPA (ICLR’26).
    • Also, the overall design is close to Training-Free GRPO (arXiv’25), which should be discussed in the paper.
  • The central thesis is not demonstrated.
    • This paper targets two problems of batch-level skill optimization: common-denominator bias and granularity mismatch.
    • However, the paper only shows the end-to-end accuracy gain, qualitative evidence of generic skill content (Fig. 2), and other analyses such as the retained-update rate and training cost.
    • So, I am not sure whether the gain really comes from resolving these problems, and whether these problems are really significant in skill optimization.
    • To address this, the authors should provide a controlled test of the thesis, a measurement of the bias itself, and a measurement of the granularity mismatch.
Questions for authors
  • Could the authors clarify which elements of GRSO are new relative to ReasoningBank, ACE, and GEPA? Please also discuss Training-Free GRPO.
  • Could the authors run an ablation that varies the reflection unit and the verification unit while reusing the same rollouts?
  • Could the authors quantify common-denominator bias, for example by categorizing learned rules as domain-specific vs. generic or format-related across GRSO, SkillOpt, and other baselines?
  • Sec. 2.3 states that batch-level bundles mix useful and harmful edits. Could the authors measure this directly?
Reviewer_RjNAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper tackles the problem of learning natural-language skills for language models. Specifically, the authors highlight two limitations of current skill learning methods: common-denominator bias and all-or-nothing acceptance. To address these issues, the authors propose GRSO, a skill optimization method inspired by GRPO that aligns the reflection and verification at the question level. The authors evaluate their method on three tasks and two language models, demonstrating that it outperforms or matches the performance of other skill-learning baselines. Additionally, the authors investigate the token cost of their method relative to baselines.

Critical strengths
  1. The paper addresses an important and timely problem. Optimizing external skills offers a promising approach to adapting language models to downstream tasks without modifying their weights. The proposed method tackles relevant challenges in this setting, particularly the extraction of useful learning signals from execution trajectories and the reliable acceptance of skill updates.
  2. The paper demonstrates substantial improvements on mathematical reasoning tasks. In particular, GRSO considerably outperforms the strongest baseline, SkillOpt, on LiveMathBench across both target models. The component ablations further suggest that the proposed acceptance, retry, and dynamic sampling mechanisms contribute to these improvements.
Critical weaknesses
  1. The paper's motivation is insufficiently grounded. The paper argues that reflecting on trajectories from different questions introduces a common-denominator bias, favoring generic instructions over question-specific knowledge. While plausible, it is unclear whether question-level reflection is generally preferable, as individual questions may provide noisy or unrepresentative learning signals, while aggregating across questions can help identify more robust and generalizable strategies. The authors briefly acknowledge this trade-off in Section 6 but do not investigate which task characteristics favor either approach. Furthermore, the evidence for common-denominator bias is limited to qualitative examples (Figure 2) and the observed plateau of SkillOpt (Figure 1), neither of which establishes that this bias causes the performance differences. A controlled ablation comparing question-level and batch-level reflection while keeping other components fixed would be important to substantiate the paper’s central motivation.
  2. The empirical evidence for general applicability of GRSO is limited. While GRSO achieves substantial improvements over SkillOpt on LiveMathBench, the gains on SearchQA and DocVQA are marginal or nonexistent. Although the authors report individual-run results in Appendix E, the main tables lack standard deviations or confidence intervals, making it difficult to assess the robustness of these smaller improvements. Furthermore, the evaluation is limited to three benchmarks and two target models, with the strongest gains concentrated on mathematical reasoning. Additional experiments across more diverse tasks, together with a systematic discussion of which task characteristics favor GRSO, would be necessary to substantiate its positioning as a general skill optimization method.
  3. The paper is missing analysis of the hyperparameter sensitivity and cost-performance trade-off. While Section 4.4 provides component ablations, the paper lacks an analysis of important hyperparameters, particularly the number of trajectories per question, the maximum number of retries, and the batch size. As these parameters directly influence both optimization performance and computational cost, a systematic sensitivity analysis would be valuable. Such experiments would also enable a more informative cost–performance comparison, showing whether GRSO’s improvements justify its additional computation and whether similar performance could be achieved with fewer rollouts.
  4. The description of the proposed method lacks clarity. Several important aspects of the proposed method are insufficiently described, particularly the cross-question merge stage. While the authors state that accepted edits are consolidated, deduplicated, and checked for conflicts, it remains unclear how these operations are performed and which criteria determine the final skill content. This is especially relevant because the merged skill is not verified after consolidation, potentially undermining the guarantees of question-level acceptance. Furthermore, Figure 3 does not clearly communicate the method’s decision flow, particularly the three outcomes of the acceptance check in Stage 4. A more precise description of these components would improve both the clarity and reproducibility of the method.
  5. The paper lacks an in-depth analysis of learned skill characteristics. Although the paper argues that question-level reflection produces more specific and useful skills, the analysis of learned skill content is largely qualitative. Beyond the examples in Figure 2 and Appendix C, the authors do not systematically investigate how the resulting skills differ from those produced by competing methods in terms of specificity, generalizability, length, or redundancy. Such an analysis would help substantiate the claimed mechanism behind GRSO’s improvements.
  6. The question-level acceptance does not guarantee generalization. GRSO accepts skill edits based solely on their performance on the question from which they were generated. While this provides a fine-grained optimization signal, it may encourage overfitting to individual training questions and does not account for potential interference with other questions. The authors rely on cross-question consolidation to address this issue, but provide no systematic analysis of whether accepted edits generalize beyond their source questions or negatively affect other examples. Such an analysis would be particularly important given that avoiding harmful updates is one of the paper’s central motivations.
Questions for authors
  1. The paper argues that batch-level reflection introduces a common-denominator bias, but aggregating information across questions may also help identify more robust and generalizable strategies. Could the authors elaborate on which task characteristics favor question-level reflection? Additionally, could they provide a controlled ablation comparing the two reflection strategies while keeping the remaining components fixed?
  2. Since proposed edits are evaluated only on their source questions, they may overfit individual examples or negatively affect performance on other questions. Could the authors investigate how frequently accepted edits improve or degrade performance on other training questions? Furthermore, could they clarify how the cross-question merge stage resolves conflicting edits and whether the resulting merged skill is validated?
  3. How sensitive is GRSO to its hyperparameters, and how does this affect the performance–cost trade-off? In particular, how do the number of trajectories per question, maximum number of retries, and batch size affect performance and computational cost? Could the authors provide a sensitivity analysis to establish whether the reported improvements can be achieved with fewer rollouts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.