acceptodds
Under review as a conference paper at ICLR 2027

Structured Self-feedback In-Context Reinforcement Learning for Test-Time Adaptation

Abstract

Unsupervised test-time self-improvement remains challenging because LLMs must identify and reuse reliable experience from their own generations without access to ground-truth or external verifiers. In-context reinforcement learning (ICRL) provides a training-free mechanism for test-time adaptation by accumulating self-generated responses and feedback in context to improve reasoning performance. However, existing ICRL methods typically derive pseudo-labels through majority voting, which treats generated responses as an unordered set and overlooks the structured evidence provided by different reasoning strategies. As a result, answers supported by only a narrow subset of reasoning processes may be indistinguishable from those consistently supported across diverse strategies. In this paper, we introduce Structured Self-feedback In-Context Reinforcement Learning (SS-ICRL), which constructs structured self-feedback from multi-strategy generations for unsupervised test-time self-improvement. Specifically, we propose a Reliability-Hill (RH) score that jointly models within-strategy agreement and cross-strategy support, enabling effective pseudo-label selection based on structured reasoning evidence. To gudide self-improvement, SS-ICRL further integrates the self-feedback into strategy-specific contextual experience across optimization rounds. Experiments across multiple mathematical reasoning benchmarks show that SS-ICRL consistently improves reasoning performance in unsupervised test-time settings without updating model parameters.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context reinforcement learning, large language models, test-time adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yWEaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores an unsupervised test-time self-improvement of LLM Reasoning to continuously enhance an LLM's reasoning capabilities during inference, without access to ground-truth answers or external verifiers, and without updating the model's parameters. The core of SS-ICRL involves leveraging the distribution of answers generated via diverse reasoning strategies to construct pseudo-labels that are more reliable than those obtained through simple majority voting. These selected reasoning trajectories, along with their associated feedback scores, are then incorporated into the model's context to facilitate multi-round in-context optimization.

Critical strengths
  1. The paper is well-organized and clearly written.

  2. The motivation of the method is interesting, considering both test-time reinforcement learning and in-context reinforcement learning.

  3. Extensive experiments are conducted to validate the effectiveness of SS-ICRL.

Critical weaknesses
  1. The main experiments focus on Qwen-family LLMs. Incorporating other LLM families would further demonstrate the generalizability of the proposed method.

  2. Please consider comparing with several related works.

[1] Sample Lottery: Unsupervised Discovery of Critical Instances for LLM Reasoning, ICLR 2026.

[2] Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning.ICML 2026.

[3] Easy Samples Are All You Need: Self-Evolving LLMs via Data-Efficient Reinforcement Learning, ACL Findings 2026.

  1. Lack of analysis regarding the independence and diversity of different reasoning strategies.
Questions for authors

Please see critical weaknesses.

2 comments

We thank the reviewer for the careful assessment and constructive feedback.

1.The main experiments focus on Qwen-family LLMs. Incorporating other LLM families would further demonstrate the generalizability of the proposed method.

We thank the reviewer for the suggestion. To assess whether the gains extend beyond the Qwen family, we evaluated SS-ICRL on AIME 2024 with Llama-3.1-8B-Instruct. The results show that SS-ICRL raises Pass@1 from 2.71% for the base model to 9.79% and outperforms both Single-ICRL and ME-ICPO.

BackboneBenchmarkBaseME-ICPOSingle-ICRLSS-ICRLΔ vs. Base
Llama-3.1-8B-InstructAIME 20242.713.235.839.79+7.08
  1. Please consider comparing with several related works. [1] Sample Lottery: Unsupervised Discovery of Critical Instances for LLM Reasoning, ICLR 2026. [2] Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning.ICML 2026. [3] Easy Samples Are All You Need: Self-Evolving LLMs via Data-Efficient Reinforcement Learning, ACL Findings 2026.

We thank the reviewer for suggesting these related works.

Comparison with SR-TTRL [2]. SR-TTRL is particularly relevant because it also constructs pseudo-labels from self-generated responses. SR-TTRL uses these pseudo-labels for test-time RL training, while our SS-ICRL uses them to update the reasoning context with fixed model parameters. To compare the quality of this feedback directly, we measured pseudo-label accuracy before the subsequent updates. We evaluated both pseudo-labeling methods under SR-TTRL’s evaluation protocol on AIME24 problems using a frozen Qwen3-4B model, with 32 candidate responses per problem and no parameter updates.

ModelDatasetSR-TTRLSS-ICRL (ours)
Qwen3-4BAIME2420.00%40.00%

With the same candidate sampling budget, our method achieved higher pseudo-label accuracy than SR-TTRL. It also scores the sampled answers directly using within-strategy agreement and cross-strategy support, avoiding the extra LLM generations that SR-TTRL uses for trajectory summarization and reflective verification.

Comparison with Sample Lottery [1] and EasyRL [3]. Sample Lottery and EasyRL focus on data-efficient reinforcement learning with parameter updates. Sample Lottery identifies informative problems without labels, but subsequently obtains ground-truth annotations for RL with verifiable rewards. EasyRL begins with supervised RL on labeled easy problems and progressively incorporates harder problems through consistency- and reflection-based pseudo-labeling. SS-ICRL instead performs label-free adaptation with a frozen model. It derives feedback from strategy-conditioned responses to the current test problem and uses that feedback to update strategy-specific contexts for further attempts on the same problem. Thus, its improvement comes from evaluating and reusing online reasoning experience, without acquiring labeled adaptation data or updating model parameters.

We will add a discussion of these works to the revised manuscript.

3. Lack of analysis regarding the independence and diversity of different reasoning strategies.

We thank the reviewer for raising this point. To quantify diversity in final-answer distributions, we analyzed first-round Qwen3-4B responses on 134 MATH-L5 problems. For each problem, we compared the four strategy groups with equal-sized random partitions of the same 32 responses. The actual strategy groups had a higher mean pairwise Jensen–Shannon divergence than the random groups (0.295 vs. 0.251, difference of 0.044, paired bootstrap 95% CI [0.029, 0.060]). This indicates that the strategy prompts elicited more distinct answer distributions in this setting. The strategies share a backbone and can produce correlated outputs. RH computes its scores directly from the observed support within and across strategies, even when their outputs overlap.

To assess the effect of the strategy set on final performance, we compared SS-ICRL using forward reasoning and representation transformation with the full four-strategy configuration on AIME 2024 with Qwen3-4B. Both configurations used 32 responses per problem per round.

ConfigurationPseudo-label selectionFinal Pass@1 (%)
Single-ICRL (one strategy)Majority voting20.99
SS-ICRL (two strategies)RH22.92
SS-ICRL (four strategies)RH27.08

With RH and the total sampling budget held fixed, expanding from these two strategies to all four improved Pass@1 by 4.16 points, even though each strategy received fewer samples.

Thanks for your rebuttal. I would maintain my positive score, please add the additional results in the revised manuscript.

Reviewer_LLJvOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SS-ICRL, a test-time adaptation framework that operates without parameter updates. It introduces the Reliability-Hill (RH) score, which jointly models within-strategy agreement and cross-strategy support to refine pseudo-label selection and guide iterative in-context optimization. Experimental evaluations in mathematical reasoning settings demonstrate consistent improvements over baseline approaches, supported by ablation studies validating key design choices.

Critical strengths

Lightweight In-Context Reinforcement Learning: The introduction of an RH-based pseudo-label selection mechanism offers a simple, parameter-free strategy to guide iterative in-context optimization, enabling effective test-time adaptation without backpropagation.

Critical weaknesses
  1. Gaps in Establishing the Connection Between Support Structure and Correctness: While Figure 1 illustrates that identical answer frequencies can stem from distinct strategy-level support structures, it does not explicitly demonstrate how these structural differences correlate with true answer correctness. Although ablation studies show that RH improves pseudo-label accuracy, the paper lacks direct analysis verifying whether individual strategy-level support distributions contain independent diagnostic signals regarding correctness.

  2. Limited Baseline Range and Ablation Variants: The pseudo-label selection experiments primarily compare standard majority voting against stripped-down variants of RH, omitting comparisons against standard model-confidence or uncertainty-based selection heuristics. Additionally, the main benchmark evaluation relies heavily on comparison against ME-ICPO and Single-ICRL, omitting key test-time adaptation baselines (such as TTRL). Broader comparisons are needed to assess the competitiveness of SS-ICRL.

  3. Restricted Task and Model Diversity: Empirical evaluations are limited to the Qwen model family and mathematical reasoning tasks. Furthermore, while seven benchmark configurations are reported, five correspond to difficulty-based subsets of MATH-500, limiting task heterogeneity. Evaluating across distinct LLM architectures and non-mathematical domains would better establish generalizability.

  4. Notation and Editorial Flaws: Several minor errors affect presentation quality. For example, "gudide" appears in the abstract, and Algorithm 1 contains inconsistent trajectory summary notation (using τ~i(t)\tilde{\tau}_i^{(t)} in line 5 versus the undefined y~i(t)\tilde{y}_i^{(t)} in line 6).

Questions for authors
  1. Ablation of Strategy Combinations across Baselines: Why does Single-ICRL rely solely on the forward reasoning strategy? Does ME-ICPO similarly employ a single fixed reasoning strategy? Under identical sampling and context-update budgets:

    • How do Single-ICRL and ME-ICPO perform when evaluated on each of the remaining three reasoning strategies individually?
    • How do they perform when utilizing all four strategies while retaining standard majority voting for pseudo-label selection?
  2. Downstream Utility and Framework Portability: What is the precise mechanism by which gains in pseudo-label selection accuracy translate into downstream in-context optimization improvements? Furthermore, can the proposed RH score be directly integrated into alternative test-time adaptation frameworks to yield comparable performance gains?

5 comments
AuthorsOpenReview ↗

Individual response (Part 1)

Thank you for your careful reading and thoughtful reviews.

W1: The paper lacks direct analysis verifying whether individual strategy-level support distributions contain independent diagnostic signals regarding correctness.

Thank you for this suggestion. We analyzed first-round Qwen3-4B responses on MATH-L5. Within each problem, we compared a correct answer and an incorrect answer whenever both received the same number of votes and each received at least two votes. The results show that even when both answers received the same number of votes, RH ranked the correct answer higher more often. This suggests that the distribution of votes across strategies provides information about correctness beyond the total vote count.

Scoring ruleCorrect answer ranked higherIncorrect answer ranked higherTie
RH68.2%27.3%4.5%

We also examined what happens when RH and majority voting select different pseudo-labels. As reported in Table 2, among the 78 cases where they selected different answers from the same responses, RH selected the correct answer in 21.79% of cases, compared with 1.28% for majority voting.

W2: Limited Baseline Range and Ablation Variants: The pseudo-label selection experiments primarily compare standard majority voting against stripped-down variants of RH, omitting comparisons against standard model-confidence or uncertainty-based selection heuristics. Additionally, the main benchmark evaluation relies heavily on comparison against ME-ICPO and Single-ICRL, omitting key test-time adaptation baselines (such as TTRL). Broader comparisons are needed to assess the competitiveness of SS-ICRL.

Comparison with Confidence-based pseudo-label selection

To extend the analysis in Table 2, we evaluated two confidence-based baselines on MATH-L5. Self-Certainty Best-of-32 selects the answer from the highest-confidence response with a valid answer. Self-Certainty + Borda Voting ranks responses by confidence and sums their rank-weighted votes for each answer. Both baselines use responses generated with our forward reasoning strategy. The table reports first-round pseudo-label accuracy (%) from the new evaluations.

BackboneSelf-Certainty Best-of-32Self-Certainty + Borda VotingRH (ours)
Qwen2.5-Math-7B47.0164.1864.93
Qwen3-4B50.7554.4865.67
Qwen3-8B52.9961.1965.67
Average50.2559.9565.42

Borda voting improved on Best-of-32 across all three backbones. At the same candidate sampling budget, RH achieved higher pseudo-label accuracy than both baselines on each backbone, averaging 65.42% versus 59.95% for Borda voting. We will include these results and the evaluation setup in the revised manuscript.

Broader test-time adaptation comparisons

We clarify that we did not include test-time training methods such as TTRL as baselines because they require updates to the model weights. To address your concerns, we added a new comparison with Self-Reflective Test-Time Reinforcement Learning (SR-TTRL)[1], which uses model reflection to assess representative reasoning trajectories and improve pseudo-label selection in TTRL. Specifically, we compared our RH pseudo-labeling mechanism with that of SR-TTRL on AIME 2024 with Qwen3-4B, using SR-TTRL’s evaluation protocol. The results show that SS-ICRL achieved 40.00% pseudo-label accuracy, compared with 20.00% for SR-TTRL’s selector.

ModelDatasetSR-TTRLSS-ICRL (ours)
Qwen3-4BAIME2420.00%40.00%

We also compared SS-ICRL with label-free adaptations of OPRO [2] and GEPA [3] for prompt-based test-time adaptation under matched task-solving rollout budgets. Across the six model–dataset settings on AIME 2024 and AMC, SS-ICRL outperformed both baselines, with average Pass@1 gains of 10.98 points over OPRO and 7.34 points over GEPA. For the full results and experimental settings, please refer to our response to Reviewer QjRR.

[1] Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning.ICML 2026. [2] Large Language Models as Optimizers. ICLR 2024. [3] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ICLR 2026.

AuthorsOpenReview ↗

Individual response (Part 2)

W3: Evaluating across distinct LLM architectures and non-mathematical domains would better establish generalizability.

We thank the reviewer for encouraging a broader evaluation. Following this suggestion, we added experiments on Llama-3.1-8B-Instruct with AIME 2024 and Qwen3-8B with GPQA, testing generalization beyond the Qwen family and applicability beyond mathematical reasoning, respectively.

ModelDatasetBaseSingle-ICRLME-ICPOSS-ICRL
Llama-3.1-8B-InstructAIME 20242.715.833.239.79
Qwen3-8BGPQA37.3638.7240.3142.08

SS-ICRL improves over the Base model in both additional settings. On AIME 2024 with Llama-3.1-8B-Instruct, Pass@1 increases from 2.71% to 9.79%. On GPQA with Qwen3-8B, it increases from 7.36% to 42.08%. These gains extend the evidence for SS-ICRL beyond the Qwen family and mathematical reasoning, without requiring labels or parameter updates during adaptation.

W4: Notation and Editorial Flaws.

Thank you for pointing out these errors. We have corrected the typo in the abstract and the inconsistent notation in Algorithm 1.

Q1: Ablation of Strategy Combinations across Baselines

Thank you for suggesting this ablation. For Single-ICRL, we chose Forward as a natural default for step-by-step mathematical reasoning. For ME-ICPO, we followed the original paper’s setup and used its default reasoning prompt as the single fixed prompt in the main comparison. To assess how strategy choice affects both baselines, we ran additional experiments on AIME 2024 with each strategy individually and with all four strategies combined, pooling answers for majority voting in the four-strategy setting.

  • Individual reasoning strategies
StrategyQwen3-4B · Single-ICRLQwen3-4B · ME-ICPOQwen2.5-Math-7B · Single-ICRL
Forward20.9913.6523.70
Backward17.8113.6519.79
Transform8.548.9624.58
Verify15.629.4827.60

The results show that the best strategy for Single-ICRL varies across backbones. Against the strongest Single-ICRL configuration on each backbone, SS-ICRL retains a Pass@1 advantage on both backbones.

  • All four strategies with majority voting

For both four-strategy baselines, we sample eight responses per strategy and pool all 32 answers for majority voting. The four-strategy extension of Single-ICRL corresponds to SS-ICRL w/o RH (MV), which retains a separate context for each strategy. The ME-ICPO variant retains its entropy-based context updates.

BackboneME-ICPO (4 strategies + MV)SS-ICRL w/o RH (MV)SS-ICRL (RH, ours)
Qwen2.5-Math-7B27.5027.7130.78
Qwen3-4B26.4621.2527.08
Qwen3-8B23.3325.0033.44

Combining all four strategies substantially improved ME-ICPO on Qwen3-4B, raising Pass@1 from 13.65% with its best individual strategy to 26.46%. With the same four strategies and sampling budget, SS-ICRL achieved the highest Pass@1 on all three backbones. Within SS-ICRL, replacing majority voting with RH improved Pass@1 by 3.07–8.44 points. We will include these comparisons in the revised manuscript.

Q2: Downstream Utility and Framework Portability: What is the precise mechanism by which gains in pseudo-label selection accuracy translate into downstream in-context optimization improvements? Furthermore, can the proposed RH score be directly integrated into alternative test-time adaptation frameworks to yield comparable performance gains?

Downstream utility. The pseudo-label determines which trajectories enter the context for the next reasoning round. When a strategy produces a trajectory whose final answer matches the pseudo-label, SS-ICRL adds the summary and feedback score of the first matching trajectory to that strategy’s context. More accurate pseudo-labels therefore reduce the chance of adding trajectories that end in an incorrect answer. In the next round, the model can build on useful intermediate steps from the retained summaries when solving the same problem. Figure 4 shows that RH-based feedback improves final estimated Pass@1 over majority-voting feedback on AIME 2024 across all three backbones.

Framework portability. In other test-time adaptation frameworks that sample and group responses by reasoning strategy, RH can directly replace majority voting at the pseudo-label selection step. For example, a framework that rewards agreement with a pseudo-label can compute those rewards using the RH-selected label and apply its existing update rule. Whether this yields comparable gains depends on both the quality of the selected pseudo-labels and how the framework uses them during adaptation.

Thanks for the rebuttal. Some of my concerns have been addressed, but I still have doubts about how to optimally judge identical answer frequencies and the efficiency of self-feedback. The improvements and contributions seem incremental. Thus, I cannot support the acceptance of this paper.

Reviewer_QjRROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SS-ICRL for test-time reasoning improvement through structured self-feedback without parameter updates. Specifically, it combines within-strategy agreement and cross-strategy support to select pseudo-labels and update strategy-specific contexts. Experiments show improvements over existing in-context adaptation baselines, with particularly strong gains on challenging mathematical reasoning tasks.

Critical strengths

I find the proposed method interesting as it introduces a new notion of consistency across reasoning strategies.

The gains are particularly pronounced on challenging benchmarks such as AIME, AMC, and higher-level MATH problems.

Critical weaknesses

The comparisons should include a broader range of methods without parameter updates, particularly majority voting and prompt/context optimization methods such as OPRO [1] and GEPA [2], under matched inference budgets and feedback access. Without these comparisons, it is difficult to assess the proposed method’s advantage over existing approaches. If the gains remain consistent against these baselines, I would be inclined to maintain my current score.

[1] Large Language Models as Optimizers. ICLR 2024.
[2] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ICLR 2026.

Questions for authors

How is a strategy’s context updated when none of its sampled responses match the selected pseudo-label?

2 comments

We thank the reviewer for the careful assessment and constructive feedback. We address the comments below.

The comparisons should include a broader range of methods without parameter updates, particularly majority voting and prompt/context optimization methods such as OPRO [1] and GEPA [2], under matched inference budgets and feedback access.

We thank the reviewer for this suggestion. We added OPRO and GEPA as parameter-free baselines to assess whether SS-ICRL offers benefits beyond prompt optimization.

Additional comparisons with OPRO and GEPA

We adapt both methods to our label-free setting by starting from a single initial instruction and constructing fixed pseudo-labels through majority voting over 32 sampled responses per problem. OPRO uses these pseudo-labels to evaluate candidate instructions, while GEPA uses them as evaluation feedback for reflective prompt optimization. Neither method accesses ground-truth answers or external verifiers during optimization. We use the same backbone for task generation and prompt optimization, align the task-solving rollout budgets, and evaluate all methods with 32 responses per problem. Prompt-proposal and reflection calls are additional to these rollout budgets.

We evaluate OPRO and GEPA on AIME 2024 and AMC using three backbone models. SS-ICRL achieves higher Pass@1 than both baselines in all six model–dataset configurations. While OPRO and GEPA refine an instruction shared across problems, SS-ICRL turns feedback on the current problem into selected reasoning experience for each strategy. Later attempts can therefore build on concrete solution steps from that problem, rather than relying solely on improvements to a general task instruction.

DatasetMethodQwen2.5-Math-7BQwen3-4BQwen3-8B
AIME 2024OPRO13.3317.6017.50
GEPA20.1017.1920.00
SS-ICRL30.7827.0833.44
AMCOPRO41.9851.4750.15
GEPA49.6652.7954.14
SS-ICRL52.7755.1258.73

How is a strategy’s context updated when none of its sampled responses match the selected pseudo-label?

If none of a strategy’s sampled responses matches the selected pseudo-label, its context remains unchanged in that round. Other strategies with matching responses update their own contexts independently. We keep each selected trajectory within the strategy that generated it so that each strategy can learn from examples of its own reasoning approach.

Thank you for your time on the rebuttal. I remain positive about the paper and will maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Manuscript Revisions and Additional Comparison with SR-TTRL

Dear Reviewers,

Following the pseudo-label comparison in our rebuttal, we ran an additional experiment to evaluate our multi-strategy RH feedback within a test-time training framework. We integrated this feedback into SR-TTRL’s GRPO training procedure and refer to the resulting variant as RH-TTRL.

We evaluated both methods on AIME 2024 with Qwen3-4B. Both trained for 10 epochs on two A100 GPUs and sampled 32 candidate responses per problem.

MethodFinal Pass@1 (%)Training wall-clock time (h)
SR-TTRL27.0817.05
RH-TTRL30.4213.18

In this setting, RH-TTRL achieved a higher final estimated Pass@1 than SR-TTRL[1] and reduced training time by 22.7%.

[1] Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning.ICML 2026.

Manuscript revisions

We have revised the manuscript to incorporate the additional results and clarifications discussed during the rebuttal.

  • Methodology. We clarified how each strategy selects a trajectory and updates its own context, including how it carries its context forward when no trajectory matches the pseudo-label.
  • Related work and experiments. We expanded the related work discussion, added comparisons with label-free adaptations of OPRO and GEPA, and extended the analysis of reasoning strategy selection and combination.
  • Appendix. We added the matched-vote analysis, comparisons with confidence-based pseudo-label selectors, additional evaluations with Llama-3.1-8B-Instruct and on GPQA, and analyses of strategy diversity and end-to-end runtime. We also added a comparison with SR-TTRL covering pseudo-label accuracy, final Pass@1, and training time.

Thank you again for your feedback and for taking the time to consider the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.