acceptodds
Under review as a conference paper at ICLR 2027

RepRSI: Can Internal Learning Progress Guide Recursive Self-Improvement?

Abstract

Recursive self-improvement relies on generated training experiences that support further learning. Immediate task performance can overlook internal changes that predict later capability acquisition. We study semantic cohesion: agreement between hidden representations of equivalent intermediate computations across contexts, relative to representations of different computations. In a controlled compositional task, semantic patching links these representations to reusable computation, and curriculum interventions show that cross-context exposure shapes cohesion and subsequent learning. Adding cohesion progress to behavioral predictors reduces mean absolute error in predicting subsequent accuracy from 19.3 to 10.1 percentage points. We introduce RepRSI, a Representation-guided Recursive Self-Improvement framework that uses curriculum-induced semantic cohesion progress to reward a curriculum-generating teacher and select student branches for continuation, while training the student with verifiable task rewards. At matched total training compute, it improves pass@32 over a teacher rewarded by target-performance gains by 13.8, 3.1, and 5.7 percentage points on Manufactoria–HAS, MATH, and HARP, respectively. Its curricula also improve fresh students, and the recursive system reaches the comparison method's final greedy performance using 65–75% of the common full budget. Cohesion progress thus provides useful feedback for recursive self-improvement beyond immediate task performance. The code is available at https://anonymous.4open.science/r/RERSI-BB36/

Primary area:
foundation or frontier models, including LLMs
Keywords:
recursive self-improvement, large language models, metacognition, curriculum learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_igvrOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper develops RepRSI, a method that uses 'semantic cohesion', i.e., the similarity between patched activations on the same concept, as an internal measure of learning progress that drives a curriculum for RL post-training. RepRSI outperforms plain RL and other curriculum strategies at short training budgets, produces a more consistent teaching policy, and generalizes better to hard subsets of data.

Critical strengths

In my view, this is a curriculum learning paper, not an RSI paper. The strengths of the method lie in its effectiveness as a curriculum.

  • The authors focus on RL post-training on challenging tasks, which is a generally interesting area for curriculum learning.
  • The choice of evaluation metrics and aggregation strategy is appropriate for the curriculum learning setting, and the baselines included are mostly relevant.
Critical weaknesses

The paper has two critical weaknesses: the RSI framing, and the curriculum comparison experiments. Reframing the paper alone is a significant revision in my view, which unfortunately leads me to recommend rejection.

Clarity and Significance

  • As it stands, this paper does not study or show RSI. The goal of RSI is to accelerate the speed of learning over time, or at least to show continued long-term improvement. The authors instead develop a method for curriculum selection, and show the typical training curves of curriculum learning — rapid improvement at low budgets.
  • Section 2 is a dense and overly concise, and a significant portion of the method's details are relegated to the appendix. There is also no discussion of why semantic cohesion, as measured, would lead to RSI, as a collapse to identical representations with differing framing may obfuscate nuanced differences between inputs. It is also not necessarily the case that predicting subsequent accuracy is the same as improving it: a model may simply see its own saturation point with no improvement across multiple curriculum strategies.
  • Likely due to the use of AI, the appendix (D.5, D.7, D.9, etc.) is overly verbose in its description of common validation protocols, and this provides unnecessary degree of detail detail obfuscates the relevant aspects of the method. In contrast, crucial aspects of the baseline implementation (D.6) are overly brief.
  • The paper does not present a limitations section, despite the results hinging on several strong assumptions.

Soundness

  • The authors do not justify why they expect semantic cohesion to lead to skill reuse, or RSI in the long term. They do not justify their choice of compute budget (92.02 GPU hours as opposed to a fixed number of training steps), and do not explore multiple training budgets or student/teacher models.
  • The computation budget comparisons do not appear to account for the outer loop of teacher training, and only presents results on the final student. However, the teacher is trained with bespoke heuristics for each task, and is not amortized over multiple students even on this one task.
  • Some of the results are less than convincing, and indicative of under-training or poor hyperparameter selection. For instance, the fail@128 set shows zero improvement with the RL training baseline, which is unexpected for RL post-training.
  • It is unclear how much of the improvement is due to the actual cohesion progress, or due to the highly specific heuristics (e.g., 'integer constants from -20 to 20' for math) that are glossed over in Appendix C.2.
  • Following on from the above, several baselines could be constructed from these heuristics, as static curricula over each (or over a linear mixture). These are not compared to in this work.
Questions for authors

Clarification requests

  • Recursive self-improvement (RSI) typically refers to the acceleration in learning speed from a self-improving system, but the results in this paper show the typical curriculum learning characteristics of rapid learning followed by saturation. Can you justify your framing of this work through the lens of RSI?
  • Section 2 is very poorly explained, likely due to heavy AI usage. In your own words, can you motivate the reason for the broad definition of a state (equation 1) and justify why you expect semantic cohesion to lead to better long-term performance?
  • How much of the improvement in can be attributed to the semantic cohesion training for the teacher model, and how much to the dataset-specific heuristics? Does your method work with an identical difficulty measure to the baseline curriculum ('reference condition' in Appendix D)?
  • The process requires several candidate curricula to be evaluated dynamically, and Figure 2 only accounts for the computational cost of final student training. What is the added computational expense of training induced by your method, including training the teacher? How does it compare to other curricula?
  • Appendix D outlines baseline implementations, but is not sufficiently detailed, especially for the 'reference conditions'. There is no discussion of the hyperparameter optimization for these baselines, and simple but effective curricula (e.g., sequence length for math) are not explored. Elaborate on the baseline construction.

Suggestions for future work

  • The authors have made substantial progress in developing a curriculum for RL-based post-training. I suggest that they reframe this as a curriculum learning paper focusing on limited budgets, and show that the method can improve with difficulty measures that are identical to the baselines (or incorporating these measures into the method).
  • If the goal is to develop a work for RSI, then the setting should be in continual learning. Can the teacher model improve not only a fresh identical student, but a trained student on a completely different problem?
14 comments
AuthorsOpenReview ↗

Short clarifications and Brief response.

Thank you very much for your careful review and for recognizing the curriculum-learning contribution of our work. We would like to clarify a few points and provide some additional context for the experiments reported in the paper.

  • “As it stands, this paper does not study or show RSI.” Thank you for raising this point. We would like to clarify that increasing the "speed of learning" was not our primary contribution. We focus on a question that we consider more fundamental to RSI: How can a model use successive rounds of training to solve problems that earlier versions could not solve on their own? This question is also central to the self-play literature that motivates our work [1, 2, 3]. To our knowledge, these works do not treat an increasing the "speed of learning over time" as a defining requirement for self-improvement.

  • There is also a curriculum-learning perspective on this question. A longstanding challenge in agentic RL is deciding what an agent should practice next. In our setting, the teacher itself learns from the student’s progress on the curricula it proposes. Each round therefore improves our basis for choosing the next curriculum: the teacher receives feedback on what helped the student learn and uses that feedback to update what it proposes next. In our view, a system that continually improves its own curriculum through this process is engaging in a form of RSI.

  • We appreciate the reviewer’s emphasis on the need to “show continued long-term improvement.” We believe our results partially meet this criterion. Across all three benchmarks, the student continues to improve after first reaching the target-grounded teacher’s final mean greedy accuracy. Figure 8 and Table 2 show these later gains over the 100-round training horizon.

    1. Manufactoria–HAS. The student reaches the comparison target at 65% of the training budget. Its greedy accuracy then rises from 49.9% to 56.5% over the remaining 35% of the budget, a further gain of 6.6 percentage points.
    2. MATH. The student reaches the comparison target at 75% of the budget. Its greedy accuracy rises from 34.9% to 37.6% over the final quarter of training, a further gain of 2.7 percentage points.
    3. HARP. The student reaches the comparison target at 65% of the budget. Its greedy accuracy then rises from 18.1% to 21.8%, a further gain of 3.7 percentage points.
  • These results establish that the gains extend into the later stages of training. Our experiments end at 100 outer rounds, so they leave open how long those gains persist and whether the teacher continues to generate more effective curricula as the student approaches a plateau. In future work, we will focus on longer training runs and evaluate teachers from successive rounds on fresh students with a common initialization. This will allow us to track both the student’s performance and the teacher’s ability to design useful curricula over time.

[1]Huang, C., Yu, W., Wang, X., Zhang, H., Li, Z., Li, R., Huang, J., Mi, H., & Yu, D. (2026). R-Zero: Self-evolving reasoning LLM from zero data. ICLR 2026.
[2]Sundaram, S., Quan, J., Kwiatkowski, A., Ahuja, K., Ollivier, Y., & Kempe, J. (2026). Teaching models to teach themselves: Reasoning at the edge of learnability. ICML 2026.
[3]Zhao, A., Wu, Y., Yue, Y., Wu, T., Xu, Q., Lin, M., Wang, S., Wu, Q., Zheng, Z., & Huang, G. (2025). Absolute Zero: Reinforced self-play reasoning with zero data. NeurIPS 2025.

AuthorsOpenReview ↗

Response for Weakness 1 and Question 1.

Weakness 1: As it stands, this paper does not study or show RSI. The goal of RSI is to accelerate the speed of learning over time, or at least to show continued long-term improvement. The authors instead develop a method for curriculum selection, and show the typical training curves of curriculum learning — rapid improvement at low budgets.
Question 1: Recursive self-improvement (RSI) typically refers to the acceleration in learning speed from a self-improving system.

  • We appreciate the reviewer’s point that evidence of continued improvement is important to the RSI interpretation. We believe our results already meet this criterion. Section 4.3 reports that “the recursive system first reaches the Target-grounded Teacher’s final mean greedy score at 65%, 75%, and 65% of the common budget on Manufactoria–HAS, MATH, and HARP, respectively.” Table 2 also shows what happens after those checkpoints:
  1. On Manufactoria–HAS, greedy accuracy rises from 49.9% to 56.5% over the final 35% of the budget.
  2. On MATH, greedy accuracy rises from 34.9% to 37.6% over the final 25% of the budget.
  3. On HARP, greedy accuracy rises from 18.1% to 21.8% over the final 35% of the budget. These results show further gains late in the 100-round runs on all three tasks. How long those gains persist remains open. In future work, we will study longer runs and evaluate teachers from successive rounds on fresh students with a common initialization. This would allow us to track whether the teacher continues to improve its curriculum even as the continuing student approaches a plateau.
AuthorsOpenReview ↗

Response for Weakness 2.

Weakness 2: There is also no discussion of why semantic cohesion, as measured, would lead to RSI, as a collapse to identical representations with differing framing may obfuscate nuanced differences between inputs. It is also not necessarily the case that predicting subsequent accuracy is the same as improving it: a model may simply see its own saturation point with no improvement across multiple curriculum strategies.

  • w2.1. “There is also no discussion of why semantic cohesion, as measured, would lead to RSI”

  • Cohesion progress gives the teacher feedback on which curricula develop intermediate computations that the student can reuse in later tasks. Section 2.2 tests reuse directly: transferring an intermediate representation into a different query increased the probability of the answer implied by that state and the new continuation by 11.2 percentage points. Section 3 explains how this signal enters the recursive loop: “RepRSI uses curriculum-induced cohesion progress to update the teacher and select a student branch for continuation.” Table 5 isolates the teacher update. With branch selection fixed, replacing target-accuracy gain with cohesion progress as the teacher reward raised fresh-student accuracy on Manufactoria–HAS from 45.2% to 56.1%. The updated teacher thus generated more effective curricula even for a student trained again from the initial checkpoint.

  • w2.2. “as a collapse to identical representations with differing framing may obfuscate nuanced differences between inputs.”

  • The cohesion measure distinguishes global collapse from increased agreement within a state, but it does not capture every meaningful distinction between inputs. Section 2.1 defines Φ as “within-state relative to between-state mean cosine similarity.” If all representations are identical, these terms are equal and Φ is zero. The observed changes also distinguish the two cases: at round 90 on the held-out MATH monitoring pool, within-state similarity increased by 0.093 while between-state similarity decreased by 0.047 (Table 9). We have added the following clarification to §2.1: “Two inputs can share an intermediate state and still require different final answers. Cohesion measures agreement at that intermediate step; preserving the distinction between subsequent operations requires a separate behavioral test.”

  • We have added a minimal-pair evaluation protocol to test whether the student preserves distinctions that change the correct answer. Starting from the example in Appendix C.2, we compare a wording-only edit with an edit that keeps the intermediate value fixed at 5/65/6 but changes the final operation from multiplication to division. The first edit tests whether the student preserves the answer across different phrasings. The second tests whether it correctly changes the answer from 55 to 5/365/36.

Minimal editInitial studentWithin-state reward (ΔC⁺)Full cohesion reward (ΔΦ)
A. Rephrase only89.4%90.6%91.0%
B. Change one input value85.2%86.4%86.8%
C. Change only the continuation79.4%80.5%80.9%
  • With full cohesion feedback, the student answered both items correctly on 91.0% of wording-only pairs and 80.9% of pairs with the same intermediate value but different continuations, compared with 90.6% and 80.5% under within-state feedback alone. Thus, the student preserved the answer across equivalent phrasings and correctly distinguished the two continuations in 80.9% of cases.
Reviewer_McEDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigate whether internal representational changes can serve as a more effective feedback signal than immediate task accuracy for curriculum based recursive self improvement in llms. It measures semantic cohension which quantifies the cosine similarity between hidden representations of equivalent itermediate computational states across dinstinct context, minus the similartity between representations of differing states.

Critical strengths
  1. The paper connects a representation diagnostic to prospective learning outcomes and actual training decisions
  2. The evaluation addresses whether the method learns useful teaching policies and whether its gains survive compute accounting.
Critical weaknesses
  1. The method needs hand-designed problem generators and predefined intermediate-state labels. This limits its use on tasks where those structures are difficult to define.
  2. The experiments all use non-thinking generation. Whether the same cohesion signal helps with thinking enabled is an important gap in the evaluation.
Questions for authors
  1. Why was thinking disabled, and have you tested whether the method still helps with thinking enabled?

  2. Can you break down the MATH and HARP results by subject, especially geometry and counting/probability? This would help show how broadly the gains extend beyond the four synthetic computation families.

  3. Appendix D2 and Table 6 mention Gemma E2B and Qwen3, but I could not find their performance results. How does the method perform on these models?

3 comments
AuthorsOpenReview ↗

Response for Question 1.

  • We used a common non-thinking setting in the original experiments to keep generation mode fixed across feedback methods. We add a thinking-enabled comparison on MATH and HARP to test whether the advantage of RepRSI persists. | Benchmark | Student thinking | Teacher greedy (%) | RepRSI greedy (%) | Greedy gain (pp) | Teacher pass@32 (%) | RepRSI pass@32 (%) | pass@32 gain (pp) | |---|---|---:|---:|---:|---:|---:|---:| | MATH | Disabled | 33.2 | 37.6 | +4.4 | 82.1 | 85.2 | +3.1 | | MATH | Enabled | 34.7 | 38.1 | +3.4 | 83.0 | 84.6 | +1.6 | | HARP | Disabled | 17.4 | 21.8 | +4.4 | 52.9 | 58.6 | +5.7 | | HARP | Enabled | 24.4 | 28.4 | +4.0 | 54.1 | 59.3 | +5.2 |

With student thinking enabled, RepRSI achieved higher greedy accuracy and pass@32 than the Target-grounded Teacher on both benchmarks. The gains were smaller than in the non-thinking runs, with the largest reduction in pass@32 on MATH.

AuthorsOpenReview ↗

Response for Question 3.

On MATH, RepRSI improves greedy accuracy from 29.2% to 30.2% and pass@32 from 72.1% to 72.2% on Gemma 4 E2B-IT. On Qwen3-4B-Instruct-2507, greedy accuracy improves from 31.6% to 32.1% and pass@32 from 78.1% to 78.3%. Both models show higher mean scores with RepRSI, although their pass@32 gains (+0.1 and +0.2 percentage points) are smaller than the +3.1-point gain on Gemma 4 E4B-IT.

BackboneBenchmarkTarget-grounded greedy (%)RepRSI greedy (%)Target-grounded pass@32 (%)RepRSI pass@32 (%)
Gemma 4 E4B-ITMATH33.237.682.185.2
Gemma 4 E4B-ITHARP17.421.852.958.6
Gemma 4 E2B-ITMATH29.230.272.172.2
Qwen3-4B-Instruct-2507MATH31.632.178.178.3
Qwen3.5-9BMATH38.639.082.983.3
Qwen3.5-9BHARP21.021.654.355.8
Qwen3.5-27BMATH51.251.889.089.1
AuthorsOpenReview ↗

Response for Question 2.

We add more experiments on break down the MATH and HARP.

BenchmarkSubjectTeacher greedy (%)RepRSI greedy (%)Greedy gain (pp)Teacher pass@32 (%)RepRSI pass@32 (%)pass@32 gain (pp)
MATHGeometry19.122.5+3.471.976.5+4.6
MATHCounting & Probability27.131.4+4.380.283.7+3.5
MATHAll subjects33.237.6+4.482.185.2+3.1
HARPGeometry9.812.8+3.039.445.5+6.1
HARPCounting & Probability13.517.3+3.848.454.5+6.1
HARPAll subjects17.421.8+4.452.958.6+5.7
Reviewer_wrmJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RepRSI, a recursive self-improvement framework guided by changes in internal representations. The method measures semantic cohesion, defined as representational agreement between equivalent intermediate computations across contexts, and uses its progress to update a curriculum-generating teacher and select student branches for continuation. Experiments on compositional reasoning, program synthesis, and mathematics report improvements in learning prediction, model performance, and compute efficiency.

Critical strengths
  1. The paper addresses an interesting question: whether internal learning progress can guide future training beyond immediate accuracy gains.
  2. The paper is generally well written and clearly organized, making the main motivation and overall approach easy to follow.
Critical weaknesses
  1. The generalizability of semantic cohesion remains unclear. Building on the representation analysis and semantic patching approach in [1], the proposed metric relies on predefined intermediate states, semantic equivalence relations, and diagnostic examples. It is unclear how this approach can be extended to open-ended reasoning tasks without explicitly defined intermediate computations or executable reference procedures.
  2. The causal role of semantic cohesion in learning improvements is not fully established. Although semantic patching and controlled curriculum interventions provide supporting evidence, the interventions also change the structure and coverage of training examples. Therefore, it remains unclear whether semantic cohesion itself captures the key mechanism behind subsequent learning gains.
  3. Comparisons with simpler curriculum signals are insufficient. Although the paper demonstrates improvements over performance-based feedback adapted from SOAR [2], Figure 4 suggests that cross-context coverage itself plays an important role in subsequent learning. Simpler measures such as intermediate-state coverage, compositional diversity, or task complexity might provide similar benefits. The unique value of hidden-state diagnostics over these simpler structural signals remains unclear.
  4. The reliability of semantic cohesion for online curriculum selection is insufficiently evaluated. Candidate curricula are ranked using limited diagnostic samples, but it is unclear whether their rankings remain consistent across independently sampled diagnostic batches. Appendix E.4 also acknowledges that candidate-ranking consistency and top-1 selection repeatability remain untested.
  5. The effectiveness of the metric may depend heavily on the diagnostic design. In mathematical reasoning, the diagnostic tasks and teacher-generated curricula share the same four predefined computation families. It is unclear whether the observed benefits generalize to other reasoning structures, diagnostic definitions, or representation layers.
  6. The experimental scope remains relatively limited. Although Appendix D.2 lists three model backbones, the main experiments focus on Gemma 4 E4B-IT, without complete system-level comparisons on other architectures or scales. Since semantic cohesion may depend on model representations, broader cross-model evaluation is needed. Moreover, mathematical evaluations primarily rely on MATH and HARP. Additional experiments on newer and independent benchmarks, such as AIME 2025 and AIME 2026, would provide stronger evidence of generalization beyond the predefined computation families.

[1] How Do Transformers Learn Implicit Reasoning?

[2] Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability.

Questions for authors

See Weakness.

5 comments
AuthorsOpenReview ↗

Response for Weakness 1.

Weakness 1: It is unclear how this approach can be extended to open-ended reasoning tasks without explicitly defined intermediate computations or executable reference procedures.

  • The Domain-Specific Diagnostics appendix described the generated diagnostics as “a separate measurement set.” Their semantic labels were “obtained from executable reference computations,” as stated in Benchmark Versions and Data Partitions. The MATH and HARP evaluation problems retained their original statements. They required no intermediate-state annotations.
  • We add comparison experiments using different solutions to the same diagnostic problem. A frozen model generated solutions from independent diagnostic problem statements. The protocol extracted intermediate results from those solutions. Human annotators paired equivalent results across solutions. A frozen model independently paired the same results. Equivalence required the same mathematical relation under the same assumptions. Either result had to support the same next step. The readout preceded each intermediate result. The change in semantic cohesion during a curriculum trial gave ΔΦ. Both conditions used ΔΦ on the same candidate curricula. The selected branches received the same continuation training.
Source of equivalence judgmentsProof quality after the curriculum trial (% of maximum score)Proof quality after common continuation (% of maximum score)Change during continuation (percentage points)Fully correct proofs after continuation (%)
Independent expert pairing40.341.0+0.719.2
Automatic pairing40.340.8+0.519.0
  • Automatic pairing had a slightly lower score after continuation. The proposed diagnostic used equivalent intermediate results identified across solutions. This construction left the sequence of steps in each target solution open.
AuthorsOpenReview ↗

Response for Weakness 2 & 3.

Weakness 2: The causal role of semantic cohesion in learning improvements is not fully established.

  • We agree. Semantic patching tested whether an intermediate representation could support a different continuation. The curriculum interventions changed cross-context exposure. This change could have influenced both semantic cohesion and subsequent learning. We used cohesion progress as a predictive signal for curriculum selection.
  • We will note in the limitations section that it remains an open question whether semantic cohesion itself contributed to the observed learning gains.

Weakness 3: Comparisons with simpler curriculum signals are insufficient.

  • We add a comparison experiments on a common pool of candidate curricula. All predictors used the same behavioral and likelihood features. We added each structural signal separately: intermediate-state coverage, cross-context coverage, compositional diversity, or task complexity. We then compared the full set of structural signals with the same set plus ΔΦ.
DatasetPredictor inputsMAE for subsequent learning gain (percentage points, lower is better)Candidate curriculum ranking accuracy (%, higher is better)
MATHBehavioral and likelihood features3.0867.4
MATHBehavioral and likelihood features + intermediate-state coverage2.575.1
MATHBehavioral and likelihood features + cross-context coverage2.278.0
MATHBehavioral and likelihood features + compositional diversity2.476.2
MATHBehavioral and likelihood features + task complexity3.069.6
MATHBehavioral and likelihood features + all structural signals1.784.4
MATHBehavioral and likelihood features + all structural signals + ΔΦ1.685.0
  • Cross-context coverage gave the best rankings among the structural signals tested individually. The combined structural signals accounted for most of the improvement over the baseline. Adding ΔΦ further reduced prediction error and modestly improved curriculum ranking accuracy.
AuthorsOpenReview ↗

Response for Weakness 4.

Weakness 4: The reliability of semantic cohesion for online curriculum selection is insufficiently evaluated.

  • We tested how consistently independent diagnostic batches selected the same curriculum. Within each round, we fixed the candidate checkpoints and recomputed ΔΦ on independently sampled diagnostic batches. We compared the resulting rankings and recorded how often the same curriculum ranked first. All candidates received the same continuation training in the comparison.
Diagnostic examples per batchRuns × evaluated rounds per runCandidates × replicas per roundIndependent batches per roundCandidate-pair ranking agreement (%)Same curriculum ranked first (%)Gap to best candidate after common continuation (MATH accuracy, percentage points)
1285 × 34 × 22075.756.51.1
2565 × 34 × 22081.068.00.8
5125 × 34 × 22085.477.20.8

With 256 diagnostic examples per batch, two independently sampled batches selected the same curriculum in 68.0% of comparisons. With 512 examples, agreement rose to 77.2%. At both batch sizes, the selected curriculum finished 0.8 percentage points behind the best candidate in MATH accuracy after common continuation training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.