acceptodds
Under review as a conference paper at ICLR 2027

CORE: Contrastive Reflection Enables Rapid Improvements in Reasoning

Abstract

Language models can use verifiable rewards to improve at a wide variety of reasoning tasks. However, both parametric (e.g. RLVR) and non-parametric (e.g. prompt optimization) approaches to doing so typically require hundreds of training samples and thousands of model rollouts, making them expensive in the best case and intractable in the worst. To address this challenge, we introduce Contrastive Reflection (CORE), a non-parametric learning algorithm that compares past reasoning traces to generate insights: short natural-language descriptions of reasoning strategies and constraints that capture differences between successful and unsuccessful problem attempts. Across four reasoning tasks, we demonstrate that CORE enables more rapid improvement than both parametric (GRPO) and non-parametric (GEPA, episodic RAG, and MemRL) methods, while using fewer rollouts. Under fixed rollout budgets with as few as five training samples, CORE achieves the strongest performance in most task–data regimes. Finally, we highlight how CORE is substantially more context-efficient than non-parametric baselines, requiring fewer prompt tokens while storing learned knowledge as compact, interpretable natural-language insights. Our results therefore suggest that distilling contrasts between successful and unsuccessful reasoning traces into abstract and useful insights can provide a more efficient and interpretable route to model self-improvement than weight updates, prompt optimization, or direct reuse of stored reasoning traces.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reasoning, self-improvement, non-parametric learning, sample efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MvGDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CORE, a non-parametric approach to adapting frozen language models. CORE compares successful and failed reasoning traces to extract reusable natural-language insights, validates them before adding them to an external memory, and retrieves them based on semantic relevance and estimated utility. The method is evaluated on four reasoning tasks, primarily using GPT-OSS-120B, with additional experiments on Qwen3.5-9B. Compared with GRPO, GEPA, episodic RAG, and MemRL, CORE achieves competitive or better performance in most low-data settings, with fewer rollouts and lower context overhead. Ablations suggest that both contrastive reflection and utility-aware retrieval contribute to the gains.

Critical strengths
  1. On GPT-OSS-120B, CORE achieves the highest mean accuracy in 9 of the 12 combinations of task and training set size. Additional results on Qwen3.5-9B provide further support for the approach. Although the methodological novelty is incremental, CORE has practical value as a way to adapt frozen models using a small number of training examples.
  2. CORE adds only about 920 context tokens per evaluation problem, compared with roughly 33,600 for episodic RAG and 32,700 for MemRL. This substantially lower context overhead is a useful advantage, especially given its competitive accuracy.
Critical weaknesses
  1. The evidence for contrastive reflection, a central contribution of the paper, is limited. In Section 5, reflection on failed traces alone improves accuracy by 23.4 percentage points, compared with 26.8 points for full CORE. The simpler variant therefore retains about 87% of the gain. Figure 4 reports only averages across tasks, without uncertainty estimates or per-task ablation results. Contrastive reflection appears to help, but it is unclear whether the additional benefit is consistent across tasks and runs.
  2. CORE validates candidate insights before adding them to memory, but there is no ablation that removes this filtering step; Appendix D only varies the number of validation attempts. The compute comparison is also incomplete: rollout counts include admission tests but exclude reflection calls, and total token usage is not reported for the ablation variants. This makes it difficult to separate the gains from contrastive reflection from those due to candidate filtering or additional computation.
Questions for authors
  1. Do the single-trace ablations use the same candidate-generation limits, admission tests, sampling strategy, and retrieval method as full CORE?
  2. Could you provide per-task ablation results and uncertainty estimates across runs? In particular, how consistent is the additional gain from contrastive reflection?
  3. What is the total token usage for each ablation variant, including reflection and admission testing?
  4. Have you evaluated a variant without admission filtering while retaining contrastive generation and utility-aware retrieval? Comparing these variants under the same compute budget would help clarify the contribution of filtering.
Reviewer_gxJ5OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CORE, a non-parametric approach to improving language model reasoning through contrastive reflection. The method maintains two external memory stores, one for successful reasoning trajectories and another for reusable natural-language insights. When the model fails to solve a problem, CORE compares the failed trajectory with a similar successful trajectory to generate candidate insights about reasoning strategies and constraints. CORE tests these insights using verifiable rewards before adding them to memory. During subsequent problem solving, CORE retrieves relevant insights based on semantic similarity and estimated utility, without updating the model parameters.

Critical strengths
  1. The paper studies an important practical problem: improving the sample and rollout efficiency of language models on reasoning tasks with verifiable rewards. The proposed framework is relatively straightforward and integrates contrastive reflection, insight admission testing, and utility-aware retrieval into a unified non-parametric learning process.

  2. The paper provides extensive experimental comparisons against both parametric and non-parametric baselines, including GRPO, GEPA, Episodic RAG, and MemRL. CORE achieves strong performance across four reasoning tasks under different training sample sizes, particularly in low-data settings.

  3. The authors provide additional experiments on a smaller model, examine training and inference costs, and conduct ablations to investigate the contributions of contrastive reflection and utility-aware retrieval.

Critical weaknesses
  1. The motivation and research insight are not sufficiently developed. The paper argues that existing approaches require substantial training data and model rollouts, and draws inspiration from human learning through contrastive reflection. However, it does not clearly establish why the lack of contrastive reflection constitutes a fundamental limitation of existing methods, or what specific mechanisms prevent them from learning efficiently from limited experience.

  2. The connection between the identified efficiency problem and the proposed solution remains largely intuitive. In my opinion, the paper would benefit from a more concrete analysis of the limitations of existing approaches, rather than primarily motivating the method through analogies to human learning.

  3. The organization and presentation of the paper could be improved. In particular, the Related Work section attempts to distinguish CORE from existing approaches, but the discussion mainly focuses on describing differences in how knowledge is stored, generated, and retrieved, without sufficiently explaining the significance of these differences or how they translate into substantive methodological contributions. This makes it difficult to identify what CORE fundamentally adds beyond existing reflection-based and memory-based approaches.

  4. The mathematical equations in the Method section are not numbered.

Questions for authors
  1. The paper motivates CORE by the high sample and rollout costs of existing learning approaches. However, it remains unclear what specific limitations of these methods make contrastive reflection a necessary or particularly suitable solution. Could the authors clarify the underlying learning bottleneck that CORE aims to address, beyond improving empirical efficiency?
Reviewer_12E7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how a frozen LLM can improve on verifiable reasoning tasks from very few training problems. The proposed method, CORE, maintains an external memory of natural-language insights: after a failure, the model contrasts the failed trace with a similar successful one to propose insights, which are admitted only if they make the model solve the originating problem, and are retrieved at test time by a utility-aware score. On Matchstick Arithmetic, MathGAP, ZebraLogic and Tower of Hanoi with gpt-oss-120b (Qwen 3.5 9B in the appendix), under matched rollout budgets, CORE outperforms GRPO, GEPA, episodic RAG and MemRL in 9 of 12 conditions while adding far fewer context tokens than the episodic baselines.

Critical strengths
  1. Learning from verifiable rewards with a handful of problems and a frozen model is an important setting, and the evaluation is more careful than most work in this space: rollout budgets are matched across methods (Table 1), three runs with SEM are reported, and Appendix C accounts for training and evaluation token costs.
  2. The design choices are well motivated, in particular verifier-gated admission of insights (Line 243-249) and the baseline-relative utility U(q)=r−bqU(q)=r-b_q (Line 155-157). However, most of these pieces exist in prior work (see weakness 1).
Critical weaknesses
  1. The main contribution has largely been done in previous work, which is neither discussed nor compared. ExpeL [1] already extracts insights from contrasting successful and failed attempts and maintains them with utility counts. HtT [2] builds a library of verified rules from training problems and keeps only those that lead to correct answers, which is the admission test of Line 243-249. On the headline claim that an in-context experience library can replace parametric RL, Training-Free GRPO [3] showed that such a library learned from about 100 problems matches RL-finetuned models, and ACE [4] showed that an evolving context playbook beats GEPA, both about a year before this submission. The remaining deltas (failure-biased sampling, neighborhood utility, UCB bonus) are reasonable engineering but not enough to carry the paper. At least ExpeL-style insights and ACE should be included as baselines.
  2. The evidence that the contrastive component drives the gain is thin. In Figure 4 the full method improves by 0.268 versus 0.234 for reflecting on the incorrect trace alone, i.e. 3 points aggregated over four tasks, while per-condition SEMs in Table 1 are often 0.02-0.10. Line 470 itself notes that all CORE variants beat GEPA, so most of the gain comes from the insight-memory framework shared with [1-4].
  3. The GRPO baseline looks mis-configured. GRPO is below the no-learning baseline in 9 of 12 cells of Table 1 (e.g. Tower of Hanoi 0.077 vs 0.179), which is expected when running about 2,000 GRPO rollouts on 5-10 problems with LoRA rank 32 and lr 1e-5 (Line 316-317). A baseline that destroys performance says nothing about parametric methods, which undermines the framing of the abstract. Please add a GRPO run with early stopping and a tuned learning rate, and consider a standard benchmark (e.g. MATH or AIME) beyond four synthetic puzzle tasks.

[1] Zhao et al. ExpeL: LLM Agents Are Experiential Learners. AAAI 2024.

[2] Zhu et al. Large Language Models can Learn Rules. arXiv 2023.

[3] Cai et al. Training-Free Group Relative Policy Optimization. arXiv 2025.

[4] Zhang et al. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. ICLR 2026.

Questions for authors
  1. Line 353 vs Table 1: Figure 2 runs baselines for 4000 rollouts while Table 1 reports 2100. Which checkpoint does Table 1 report?
  2. Table 1 caption: "mean ± SEM across completed runs". Did some runs not complete?
  3. Figure 4: Does the "incorrect trace only" variant keep admission testing and utility-aware retrieval? Please report per-task numbers with SEM.
  4. Line 186-199: ZZ, λ\lambda, β\beta, ϵmix\epsilon_{mix} and the number of candidate insights per reflection are never given. Please add a hyperparameter table.
  5. Line 246: With nadm=1n_{adm}=1 and δ=0\delta=0, a useless insight on a problem with bq=0.5b_q=0.5 is admitted half the time. Appendix D tests this only on Matchstick. How often does it happen on Tower of Hanoi and MathGAP?
  6. Line 405: Episodic RAG adds 33k tokens because entire traces are put in context. Would a variant that stores only final solutions be a fairer comparison for Sec 4.4?
  7. Table 3: On Qwen 3.5 9B, GEPA beats CORE in 4 of 12 conditions (e.g. Matchstick-100 95.0 vs 84.0, MathGAP-100 86.3 vs 79.3). Please soften Line 21-22.
  8. Line 100 attributes GRPO to Cobbe et al. 2021, which is the verifier paper.
  9. Typo:
    • Line 262: experiments(Agarwal → experiments (Agarwal
    • Line 317: (Lab, 2025) → (Thinking Machines Lab, 2025)
    • Line 644: Core → CORE

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.