- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper explores an unsupervised test-time self-improvement of LLM Reasoning to continuously enhance an LLM's reasoning capabilities during inference, without access to ground-truth answers or external verifiers, and without updating the model's parameters. The core of SS-ICRL involves leveraging the distribution of answers generated via diverse reasoning strategies to construct pseudo-labels that are more reliable than those obtained through simple majority voting. These selected reasoning trajectories, along with their associated feedback scores, are then incorporated into the model's context to facilitate multi-round in-context optimization.
Critical strengths
-
The paper is well-organized and clearly written.
-
The motivation of the method is interesting, considering both test-time reinforcement learning and in-context reinforcement learning.
-
Extensive experiments are conducted to validate the effectiveness of SS-ICRL.
Critical weaknesses
-
The main experiments focus on Qwen-family LLMs. Incorporating other LLM families would further demonstrate the generalizability of the proposed method.
-
Please consider comparing with several related works.
[1] Sample Lottery: Unsupervised Discovery of Critical Instances for LLM Reasoning, ICLR 2026.
[2] Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning.ICML 2026.
[3] Easy Samples Are All You Need: Self-Evolving LLMs via Data-Efficient Reinforcement Learning, ACL Findings 2026.
- Lack of analysis regarding the independence and diversity of different reasoning strategies.
Questions for authors
Please see critical weaknesses.
2 comments
Thanks for your rebuttal. I would maintain my positive score, please add the additional results in the revised manuscript.
We thank the reviewer for the careful assessment and constructive feedback.
We thank the reviewer for the suggestion. To assess whether the gains extend beyond the Qwen family, we evaluated SS-ICRL on AIME 2024 with Llama-3.1-8B-Instruct. The results show that SS-ICRL raises Pass@1 from 2.71% for the base model to 9.79% and outperforms both Single-ICRL and ME-ICPO.
We thank the reviewer for suggesting these related works.
Comparison with SR-TTRL [2]. SR-TTRL is particularly relevant because it also constructs pseudo-labels from self-generated responses. SR-TTRL uses these pseudo-labels for test-time RL training, while our SS-ICRL uses them to update the reasoning context with fixed model parameters. To compare the quality of this feedback directly, we measured pseudo-label accuracy before the subsequent updates. We evaluated both pseudo-labeling methods under SR-TTRL’s evaluation protocol on AIME24 problems using a frozen Qwen3-4B model, with 32 candidate responses per problem and no parameter updates.
With the same candidate sampling budget, our method achieved higher pseudo-label accuracy than SR-TTRL. It also scores the sampled answers directly using within-strategy agreement and cross-strategy support, avoiding the extra LLM generations that SR-TTRL uses for trajectory summarization and reflective verification.
Comparison with Sample Lottery [1] and EasyRL [3]. Sample Lottery and EasyRL focus on data-efficient reinforcement learning with parameter updates. Sample Lottery identifies informative problems without labels, but subsequently obtains ground-truth annotations for RL with verifiable rewards. EasyRL begins with supervised RL on labeled easy problems and progressively incorporates harder problems through consistency- and reflection-based pseudo-labeling. SS-ICRL instead performs label-free adaptation with a frozen model. It derives feedback from strategy-conditioned responses to the current test problem and uses that feedback to update strategy-specific contexts for further attempts on the same problem. Thus, its improvement comes from evaluating and reusing online reasoning experience, without acquiring labeled adaptation data or updating model parameters.
We will add a discussion of these works to the revised manuscript.
We thank the reviewer for raising this point. To quantify diversity in final-answer distributions, we analyzed first-round Qwen3-4B responses on 134 MATH-L5 problems. For each problem, we compared the four strategy groups with equal-sized random partitions of the same 32 responses. The actual strategy groups had a higher mean pairwise Jensen–Shannon divergence than the random groups (0.295 vs. 0.251, difference of 0.044, paired bootstrap 95% CI [0.029, 0.060]). This indicates that the strategy prompts elicited more distinct answer distributions in this setting. The strategies share a backbone and can produce correlated outputs. RH computes its scores directly from the observed support within and across strategies, even when their outputs overlap.
To assess the effect of the strategy set on final performance, we compared SS-ICRL using forward reasoning and representation transformation with the full four-strategy configuration on AIME 2024 with Qwen3-4B. Both configurations used 32 responses per problem per round.
With RH and the total sampling budget held fixed, expanding from these two strategies to all four improved Pass@1 by 4.16 points, even though each strategy received fewer samples.