- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper develops RepRSI, a method that uses 'semantic cohesion', i.e., the similarity between patched activations on the same concept, as an internal measure of learning progress that drives a curriculum for RL post-training. RepRSI outperforms plain RL and other curriculum strategies at short training budgets, produces a more consistent teaching policy, and generalizes better to hard subsets of data.
Critical strengths
In my view, this is a curriculum learning paper, not an RSI paper. The strengths of the method lie in its effectiveness as a curriculum.
- The authors focus on RL post-training on challenging tasks, which is a generally interesting area for curriculum learning.
- The choice of evaluation metrics and aggregation strategy is appropriate for the curriculum learning setting, and the baselines included are mostly relevant.
Critical weaknesses
The paper has two critical weaknesses: the RSI framing, and the curriculum comparison experiments. Reframing the paper alone is a significant revision in my view, which unfortunately leads me to recommend rejection.
Clarity and Significance
- As it stands, this paper does not study or show RSI. The goal of RSI is to accelerate the speed of learning over time, or at least to show continued long-term improvement. The authors instead develop a method for curriculum selection, and show the typical training curves of curriculum learning — rapid improvement at low budgets.
- Section 2 is a dense and overly concise, and a significant portion of the method's details are relegated to the appendix. There is also no discussion of why semantic cohesion, as measured, would lead to RSI, as a collapse to identical representations with differing framing may obfuscate nuanced differences between inputs. It is also not necessarily the case that predicting subsequent accuracy is the same as improving it: a model may simply see its own saturation point with no improvement across multiple curriculum strategies.
- Likely due to the use of AI, the appendix (D.5, D.7, D.9, etc.) is overly verbose in its description of common validation protocols, and this provides unnecessary degree of detail detail obfuscates the relevant aspects of the method. In contrast, crucial aspects of the baseline implementation (D.6) are overly brief.
- The paper does not present a limitations section, despite the results hinging on several strong assumptions.
Soundness
- The authors do not justify why they expect semantic cohesion to lead to skill reuse, or RSI in the long term. They do not justify their choice of compute budget (92.02 GPU hours as opposed to a fixed number of training steps), and do not explore multiple training budgets or student/teacher models.
- The computation budget comparisons do not appear to account for the outer loop of teacher training, and only presents results on the final student. However, the teacher is trained with bespoke heuristics for each task, and is not amortized over multiple students even on this one task.
- Some of the results are less than convincing, and indicative of under-training or poor hyperparameter selection. For instance, the fail@128 set shows zero improvement with the RL training baseline, which is unexpected for RL post-training.
- It is unclear how much of the improvement is due to the actual cohesion progress, or due to the highly specific heuristics (e.g., 'integer constants from -20 to 20' for math) that are glossed over in Appendix C.2.
- Following on from the above, several baselines could be constructed from these heuristics, as static curricula over each (or over a linear mixture). These are not compared to in this work.
Questions for authors
Clarification requests
- Recursive self-improvement (RSI) typically refers to the acceleration in learning speed from a self-improving system, but the results in this paper show the typical curriculum learning characteristics of rapid learning followed by saturation. Can you justify your framing of this work through the lens of RSI?
- Section 2 is very poorly explained, likely due to heavy AI usage. In your own words, can you motivate the reason for the broad definition of a state (equation 1) and justify why you expect semantic cohesion to lead to better long-term performance?
- How much of the improvement in can be attributed to the semantic cohesion training for the teacher model, and how much to the dataset-specific heuristics? Does your method work with an identical difficulty measure to the baseline curriculum ('reference condition' in Appendix D)?
- The process requires several candidate curricula to be evaluated dynamically, and Figure 2 only accounts for the computational cost of final student training. What is the added computational expense of training induced by your method, including training the teacher? How does it compare to other curricula?
- Appendix D outlines baseline implementations, but is not sufficiently detailed, especially for the 'reference conditions'. There is no discussion of the hyperparameter optimization for these baselines, and simple but effective curricula (e.g., sequence length for math) are not explored. Elaborate on the baseline construction.
Suggestions for future work
- The authors have made substantial progress in developing a curriculum for RL-based post-training. I suggest that they reframe this as a curriculum learning paper focusing on limited budgets, and show that the method can improve with difficulty measures that are identical to the baselines (or incorporating these measures into the method).
- If the goal is to develop a work for RSI, then the setting should be in continual learning. Can the teacher model improve not only a fresh identical student, but a trained student on a completely different problem?
14 comments
Response for Weakness 1 and Question 1.
Weakness 1: As it stands, this paper does not study or show RSI. The goal of RSI is to accelerate the speed of learning over time, or at least to show continued long-term improvement. The authors instead develop a method for curriculum selection, and show the typical training curves of curriculum learning — rapid improvement at low budgets.
Question 1: Recursive self-improvement (RSI) typically refers to the acceleration in learning speed from a self-improving system.
- We appreciate the reviewer’s point that evidence of continued improvement is important to the RSI interpretation. We believe our results already meet this criterion. Section 4.3 reports that “the recursive system first reaches the Target-grounded Teacher’s final mean greedy score at 65%, 75%, and 65% of the common budget on Manufactoria–HAS, MATH, and HARP, respectively.” Table 2 also shows what happens after those checkpoints:
- On Manufactoria–HAS, greedy accuracy rises from 49.9% to 56.5% over the final 35% of the budget.
- On MATH, greedy accuracy rises from 34.9% to 37.6% over the final 25% of the budget.
- On HARP, greedy accuracy rises from 18.1% to 21.8% over the final 35% of the budget. These results show further gains late in the 100-round runs on all three tasks. How long those gains persist remains open. In future work, we will study longer runs and evaluate teachers from successive rounds on fresh students with a common initialization. This would allow us to track whether the teacher continues to improve its curriculum even as the continuing student approaches a plateau.
Response for Weakness 2.
Weakness 2: There is also no discussion of why semantic cohesion, as measured, would lead to RSI, as a collapse to identical representations with differing framing may obfuscate nuanced differences between inputs. It is also not necessarily the case that predicting subsequent accuracy is the same as improving it: a model may simply see its own saturation point with no improvement across multiple curriculum strategies.
-
w2.1. “There is also no discussion of why semantic cohesion, as measured, would lead to RSI”
-
Cohesion progress gives the teacher feedback on which curricula develop intermediate computations that the student can reuse in later tasks. Section 2.2 tests reuse directly: transferring an intermediate representation into a different query increased the probability of the answer implied by that state and the new continuation by 11.2 percentage points. Section 3 explains how this signal enters the recursive loop: “RepRSI uses curriculum-induced cohesion progress to update the teacher and select a student branch for continuation.” Table 5 isolates the teacher update. With branch selection fixed, replacing target-accuracy gain with cohesion progress as the teacher reward raised fresh-student accuracy on Manufactoria–HAS from 45.2% to 56.1%. The updated teacher thus generated more effective curricula even for a student trained again from the initial checkpoint.
-
w2.2. “as a collapse to identical representations with differing framing may obfuscate nuanced differences between inputs.”
-
The cohesion measure distinguishes global collapse from increased agreement within a state, but it does not capture every meaningful distinction between inputs. Section 2.1 defines Φ as “within-state relative to between-state mean cosine similarity.” If all representations are identical, these terms are equal and Φ is zero. The observed changes also distinguish the two cases: at round 90 on the held-out MATH monitoring pool, within-state similarity increased by 0.093 while between-state similarity decreased by 0.047 (Table 9). We have added the following clarification to §2.1: “Two inputs can share an intermediate state and still require different final answers. Cohesion measures agreement at that intermediate step; preserving the distinction between subsequent operations requires a separate behavioral test.”
-
We have added a minimal-pair evaluation protocol to test whether the student preserves distinctions that change the correct answer. Starting from the example in Appendix C.2, we compare a wording-only edit with an edit that keeps the intermediate value fixed at but changes the final operation from multiplication to division. The first edit tests whether the student preserves the answer across different phrasings. The second tests whether it correctly changes the answer from to .
| Minimal edit | Initial student | Within-state reward (ΔC⁺) | Full cohesion reward (ΔΦ) |
|---|---|---|---|
| A. Rephrase only | 89.4% | 90.6% | 91.0% |
| B. Change one input value | 85.2% | 86.4% | 86.8% |
| C. Change only the continuation | 79.4% | 80.5% | 80.9% |
- With full cohesion feedback, the student answered both items correctly on 91.0% of wording-only pairs and 80.9% of pairs with the same intermediate value but different continuations, compared with 90.6% and 80.5% under within-state feedback alone. Thus, the student preserved the answer across equivalent phrasings and correctly distinguished the two continuations in 80.9% of cases.
Short clarifications and Brief response.
Thank you very much for your careful review and for recognizing the curriculum-learning contribution of our work. We would like to clarify a few points and provide some additional context for the experiments reported in the paper.
“As it stands, this paper does not study or show RSI.” Thank you for raising this point. We would like to clarify that increasing the "speed of learning" was not our primary contribution. We focus on a question that we consider more fundamental to RSI: How can a model use successive rounds of training to solve problems that earlier versions could not solve on their own? This question is also central to the self-play literature that motivates our work [1, 2, 3]. To our knowledge, these works do not treat an increasing the "speed of learning over time" as a defining requirement for self-improvement.
There is also a curriculum-learning perspective on this question. A longstanding challenge in agentic RL is deciding what an agent should practice next. In our setting, the teacher itself learns from the student’s progress on the curricula it proposes. Each round therefore improves our basis for choosing the next curriculum: the teacher receives feedback on what helped the student learn and uses that feedback to update what it proposes next. In our view, a system that continually improves its own curriculum through this process is engaging in a form of RSI.
We appreciate the reviewer’s emphasis on the need to “show continued long-term improvement.” We believe our results partially meet this criterion. Across all three benchmarks, the student continues to improve after first reaching the target-grounded teacher’s final mean greedy accuracy. Figure 8 and Table 2 show these later gains over the 100-round training horizon.
These results establish that the gains extend into the later stages of training. Our experiments end at 100 outer rounds, so they leave open how long those gains persist and whether the teacher continues to generate more effective curricula as the student approaches a plateau. In future work, we will focus on longer training runs and evaluate teachers from successive rounds on fresh students with a common initialization. This will allow us to track both the student’s performance and the teacher’s ability to design useful curricula over time.
[1]Huang, C., Yu, W., Wang, X., Zhang, H., Li, Z., Li, R., Huang, J., Mi, H., & Yu, D. (2026). R-Zero: Self-evolving reasoning LLM from zero data. ICLR 2026.
[2]Sundaram, S., Quan, J., Kwiatkowski, A., Ahuja, K., Ollivier, Y., & Kempe, J. (2026). Teaching models to teach themselves: Reasoning at the edge of learnability. ICML 2026.
[3]Zhao, A., Wu, Y., Yue, Y., Wu, T., Xu, Q., Lin, M., Wang, S., Wu, Q., Zheng, Z., & Huang, G. (2025). Absolute Zero: Reinforced self-play reasoning with zero data. NeurIPS 2025.