acceptodds
Withdrawn from ICLR 2027

Self-Agent System for Reflective Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reasoning reliability is critical as LLM agents become increasingly long-horizon, since local errors can silently accumulate and propagate across steps, yet the reliability of evolving reasoning trajectories remains underexplored. To address this, we introduce the self-agent perspective: an LLM-based agent treats its own evolving reasoning trajectory as the object of perception, deliberation, and action. Specifically, we realize this perspective by internalizing the agent loop over the model's own reasoning process. A tiny learned monitor acts as perception, identifying prefixes that are likely to derail later reasoning from past model-generated conversations and final-answer feedback. The frozen LLM itself then serves as deliberation, re-examining the suspect reasoning in a separate context rather than blindly continuing the flawed trajectory. A conservative commit gate provides action, writing a repair back only when it is specific and reliable. In this sense, self-agent turns a fixed LLM from a one-way generator into an online controller of its own reasoning trajectory, without retraining the generator, annotating intermediate steps, or relying on stronger planner/controller models or specialized expert agents. Comprehensive experiments on mainstream mathematical reasoning benchmarks show that Self-Agent provides plug-and-play gains across base models, improving long-horizon reasoning and recovering failing trajectories with only modest additional computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM agents, long-horizon reasoning, self-correction, inference-time reasoning, reasoning trajectory monitoring

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_6XRVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author of this paper studies the problem that in long-chain reasoning with LLMs, when an issue occurs at a certain step, it cannot be corrected in time, and the error propagates further and further, leading to an incorrect result. Therefore, in order to truly supervise in real time during reasoning whether erroneous reasoning occurs, the author proposes their own method, namely, when the model generates reasoning, there is a monitoring module (a 2-layer MLP with about 0.33M parameters) that supervises in real time. The monitoring module is divided into three parts: a monitor used to diagnose whether the current reasoning will lead to an incorrect result; a diagnostician (the same base model) that opens a separate context to try to repair the current step, avoiding the original model's preconceptions; and a commit gate (a conservative rule-based mechanism) used to determine whether the diagnostician's repair is reasonable. In terms of experiments, the method is mainly validated in two domains: mathematical reasoning and code generation. The mathematical tasks include GSM8K, MATH-500, and AIME 2025, and the code tasks include MBPP, with experiments conducted on models of different scales such as Qwen3-4B/8B/14B/32B. The author compares Self-Agent with methods such as Vanilla, Self-Check, and Self-Consistency, and further validates the effectiveness of the method through budget-matched Self-Consistency, Rescue/Harm analysis, and ablation experiments on the Monitor, independent Diagnostic Context, and Commit Gate. The experimental results show that Self-Agent can improve the model's final accuracy under limited additional inference overhead, and successfully rescue some results from originally incorrect reasoning trajectories, while keeping the damage to originally correct trajectories at a relatively low level; however, its benefits diminish as the base model scale increases, and without considering computational cost, the final accuracy of Self-Consistency is still higher.

Critical strengths

The method proposed in this paper for real-time monitoring of long-chain LLM reasoning is implemented in a lightweight manner. Apart from training a lightweight MLP, it requires no modification to the base model, and at the same time it improves the performance of the base model. The idea of lightweight design is highly valuable.

Critical weaknesses

The innovation is limited, and the performance improvement is insufficient. Previous work, such as Reflective Confidence: Correcting Reasoning Flaws via Online Self-Correction (ICASSP 2026), has already explored online monitoring during the reasoning process, using the model's own generated probabilities as confidence signals to trigger self-correction. In contrast, this work introduces an additional 0.33M-parameter MLP monitor, but it does not demonstrate sufficiently significant improvements in either methodology or lightweight design. In addition, most of the experimental baselines are based on relatively outdated methods from 2022 to 2023, and the proposed method does not consistently achieve state-of-the-art performance. Considering the additional monitoring and diagnostic components, the relatively limited performance improvement makes it difficult to assess whether the added complexity is sufficiently justified.

Questions for authors
  1. Can comparisons be provided with more recent and up-to-date real-time monitoring self-correction methods?
  2. Can generalization experiments on zero-shot transfer be provided to demonstrate the generalization ability of the lightweight method?
Reviewer_xG6GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Self-Agent, an inference-time wrapper that monitors a frozen LLM's own reasoning trajectory and repairs it online. It has three parts:

(1) Reflection monitor. A small MLP (one hidden layer, about 0.33M parameters) is trained on the frozen model's last hidden state at a prefix to predict "continuation reliability". That reliability is the probability that continuing from the prefix reaches a correct final answer, estimated by sampling continuations from the frozen model. Prefixes with low estimated reliability become binary high-risk labels; ambiguous ones are dropped. The monitor is trained on rollouts from each base model on the GSM8K/MATH training splits, and on MBPP's 374 train / 90 val split.

(2) Diagnostic module. When the monitor's risk score exceeds a threshold γ, the same frozen model, placed in a separate context, inspects the suspect local region and returns a structured proposal (decision, confidence, corrected fact, rewrite scope).

(3) Commit gate. A repair is written back to the live trajectory only if it is high-confidence, non-empty and locally scoped. Otherwise generation continues unchanged.

The intervention policy is formalized as a thresholded rule, and γ is selected on validation data to maximize rescued minus harmed trajectories. Each trajectory gets at most one intervention, so overhead is bounded at 1.5–2.4× tokens.

Validation. Qwen3 models at 4B/8B/14B/32B on GSM8K, MATH-500 (thinking mode disabled to leave headroom), AIME 2025 (30 problems) and MBPP. Reported gains:

  • GSM8K: +3.5 / +0.7 / +0.1 / −0.2 points;
  • MATH-500: +3.2 / +3.6 / +1.4 / +0.8;
  • AIME: +3.3 / +6.7 / +3.3 / +3.3;
  • MBPP: +4.8 and +2.6 Pass@1.

Harm rates are low (0.2–5.6%). Ablations cover hidden states versus 13 token-level uncertainty features (AUROC 0.793/0.832 vs 0.733/0.684), gate variants, separate-context versus in-context self-checking (self-checking turns negative at k=30/40), monitor depth/width, pooled versus dataset-specific monitors, and 8B→14B monitor transfer. Budget-matched self-consistency (SC@3/SC@5 applied to a random subset so that expected token cost matches) is used as the main efficiency baseline.

Critical strengths
  1. A clean, cheap, and honestly-bounded mechanism, with the right ablations to support its design choices. The framing of "monitor the prefix, not the token" is simple, and the paper substantiates each design decision rather than asserting it:
    • hidden states beat 13 uncertainty features by a wide margin (0.793 vs 0.733 AUROC on 4B; 0.832 vs 0.684 on 8B), which is the key empirical claim for the monitor;
    • separate-context diagnosis beats in-context self-checking, and the comparison is run at matched trigger points on the same prefixes, so the difference is attributable to the context switch rather than to a budget difference (self-checking goes from +1.1 to −2.6 as k grows, while Self-Agent stays positive);
    • the cost is genuinely bounded at one intervention, 1.5–2.4× tokens, reported per operating point. Together these make the paper a useful, reproducible data point on where reliability gains come from, which is more than most inference-time-refinement papers establish.
  2. The efficiency comparison is done properly, and the harm accounting is unusually honest. Budget-matched SC is constructed by routing a random subset to SC@3/@5 and averaging over 20,000 assignments, so the baseline gets the same expected tokens rather than a strawman. The paper reports rescued and harmed rates for every cell, and reports the full-budget SC results in the appendix instead of hiding them. That transparency is worth crediting: it shows Self-Agent at 1.55× tokens reaching 89.5 on 4B GSM8K, versus 92.3 for SC@5 at 5.07×, which is a real efficiency argument even though full-budget SC wins on raw accuracy.
Critical weaknesses
  1. The central claim of "plug-and-play, no retraining" is overstated, and this affects how the contribution should be valued. The introduction motivates the work partly by closed-source LLM APIs, where the base model cannot be modified. But the method requires (i) the last hidden state of the base model, which hosted APIs do not expose, and (ii) a monitor trained on rollouts from that specific model on GSM8K/MATH training splits, plus sampled continuations to estimate Q(p). Every new base model therefore needs its own rollout-and-probe pipeline, which is a training procedure even if it is a small one.

The paper's own transfer experiment confirms this: moving the 8B monitor to 14B yields +1.0 point on MATH-500 at top-20, only on the easy split, and only via a re-encoding hack that runs 8B forward passes on 14B rollouts. So the monitor is not model-agnostic, and the "no retraining" claim holds only for the generator.

This matters for the assessment because the paper's positioning against training-based and multi-agent approaches rests on this property.

  1. The empirical support is thin where it matters most: no variance estimates on the main table, and the operating point appears to be chosen on the evaluation objective. (a) Table 1 has no confidence intervals or seed variation. AIME is 30 problems, so the reported +3.3 / +6.7 / +3.3 / +3.3 correspond to 1, 2, 1 and 1 problems respectively, and harm rates of 5.6% and 5.3% for 4B/8B mean one damaged correct answer out of 18. These rows cannot distinguish the method from noise.

(b) The threshold γ is selected by maximizing rescued minus harmed (Eq. 2), i.e. exactly the quantity that is then reported as the result, and the paper does not say which data this selection used or whether it is disjoint from the reported test sets. B.1 further states that the best threshold differs by model and benchmark, and Table 1 never reports which k ∈ {20, 30, 40} produced each row. As written, a reader cannot rule out that the operating point was tuned on the evaluation data, in which case every delta is optimistically biased.

(c) The gains vanish at scale in a way the abstract does not reflect: 14B GSM8K +0.1, 32B GSM8K −0.2, 32B MATH-500 +0.8. The abstract's "plug-and-play gains across base models" is true only for the 4B/8B GSM8K/MATH cells.

(d) There is also an attribution gap. The monitor alone could support a cheaper intervention — resample the continuation from the risky prefix and keep the monitor's preferred outcome — which uses the same probe and no diagnostic call. Without that control, the paper cannot separate "the monitor finds the right place" from "an extra LLM call at that place helps", even though the gate and diagnostic ablations are otherwise thorough.

Questions for authors
  1. Operating-point protocol. For each row of Table 1, was k (or γ) selected on a held-out validation split disjoint from the reported test set, and which k was used? Please report the selected k per model/dataset, and the size of the validation split used in Eq. 2. If selection used the evaluation data, please re-run with a frozen threshold and report the resulting deltas.
  2. Variance. Can you report at least 3 seeds (or bootstrap intervals over problems) for Table 1 and Table 2? For AIME, please report the number of changed problems and a paired test, since 1 problem = 3.3 points at N = 30.
  3. Missing control. How does Self-Agent compare against a monitor-only intervention that, at the same trigger point and the same token budget, resamples the continuation from the risky prefix (and optionally keeps the result whose monitor score is lower) instead of calling the diagnostic module? This isolates the contribution of diagnosis from the contribution of localization.
  4. Honest scope. Since the monitor requires hidden states and per-model rollouts, do you still claim compatibility with closed-source API models? If the intended deployment is open-weight models with a per-model probe, please state the amortized cost of building the probe for a new backbone, and revise the framing accordingly.
  5. Benchmark consistency. Thinking mode is on for GSM8K/AIME and off for MATH-500/MBPP. Do the relative gains survive under a single decoding regime across benchmarks?
  6. Will code for the monitor, the diagnostic prompts and the gate be released, and is the gate's acceptance rule (confidence threshold, locality bound) fixed across all models or tuned per model?
Reviewer_wEynOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper demonstrates a method for modifying a reasoning language model's chain of thought as it reasons, pairing a small MLP/LR-based monitor with the RLM as a rewriter to correct errors during deployment. The authors demonstrate that this paired method achieves gains in accuracy on single-turn mathematics and coding problems with a lower cost relative to other consistency-based self-correction baselines and analyze various choices in terms of threshold setting and monitor training for their role in facilitating successful responses.

Critical strengths

(1) The choice of a lightweight, activation- or uncertainty-based classifier to detect potential errors is an interesting approach. It follows from a lot of recent success in using linear probes for settings where supervised learning is feasible. The utility for production systems is self-evident, and the authors do a good job defending why activations are an especially helpful learning signal. This is the largest positive signal, as satellite-like classifiers orbiting around a main policy seem like they would be useful for improvement and rewiring. I wonder how much this could be incorporated into training time itself, as opposed to deployment in production, but the use of a monitor + threshold seems reasonable and intriguing. It is also no

(2) The improvements seem somewhat real, especially when looking at hardness-stratified and qualitative examples. I appreciate that the authors included these. The results are most substantial for AIME and at smaller scales. These are the substance which make me view the paper in a more positive light, although my more nuanced evaluation on this can be understood from the weaknesses + questions.

(3) The paper is simple, clear, well-explained and easy to follow. The main findings, narrative structure, intuitions from authors and subsequent analyses follow logically from their precursors. This is a substantial positive -- refreshing with respect to my reviewing pool -- but it does not change my views on the merits of the research itself.

This is good work, but it seems more fit for a workshop at this time. In my opinion, taking the time to incorporate more modern baselines, envisioning a stronger theory of impact/change for this intervention for 2027, and doing more careful study of the relationship between monitors and tasks would take this from workshop-ready to award-at-conference ready. Good job following a simple, clear, and novel lead.

Critical weaknesses

(1) Relationship to existing work: the authors neglect to cite Recursive Language Models, which seems like a glaring omission. That work also provided some foundational framing for model context as itself an object of manipulation, enabling other copies to conduct refinement and task decomposition for better efficiency, rewriting and accuracy. As this paper is quite popular and did inspire some production-level implementations, and it references directly the authors' primary framing of Self-Agent as using reasoning traces for objects of study, it seems essential to not only reference this in Related Works but also to use it directly as a baseline. This is the strongest factor limiting my recommendation for acceptance -- the novelty of the activation-based probes are a second-order inclusion under the current framing, but more recent works already take up a lot of the calls to action in introduction's normative frame. These include other, lower-profile papers that would also be good compare against such as this study of probe insufficiency for reasoning, or the use of probes in AI safety for lie detection or scheming. The current related works are quite old, and definitely need a refresh. As it stands, I struggle to characterize why I think the contribution in this work is novel in its treatment of reasoning context as manipulable, instead it is reduced to probing for uncertainty. I am sure there are other papers that are more similar, the methods of which I invite the authors to look for these and either describe in writing or compare against in experiments. See Question 1.

(2) Insufficient proof of generalization across tasks, especially out of distribution. This is the largest substantive problem I take away from the paper. After reading, I am convinced that we can use supervised labels in-distribution to flag potentially incorrect reasoning traces, and we can also maybe use uncertainty signals for the same. But these monitors are trained on single-turn math tasks which are quite saturated. Moreover, each monitor is always trained in-distribution on these already-saturated benchmarks (even the Appendix pooled settings are in-distribution, since testing is done on evals in the pool). Perhaps I am wrong, but I don't see an "out-of-distribution" evaluation even in the most lenient sense where train-GSM8K/test-MATH is evaluated -- to say nothing about generalization across verifiable tasks (i.e. MBPP-to-MATH) or to non-verifiable tasks! This relates to a more fundamental problem with the proposal: does it scale? Where does it fit within a scaling paradigm? If the problem is that models sometimes make local mistakes, it seems like those mistakes are becoming less frequent as a function of overall model strength, and that tomorrow's models are very unlikely to fall prey to small arithmetic errors. What role would such monitors play in more advanced settings, perhaps ones where ground-truth labels for reasoning rollouts are not available? To change my mind on this, I'd like to see at minimum some lenient-OOD evaluations, re-using your monitors for other settings like GPQA, IFEval or MMLU-Pro, even as much as showing their OOD AUROC on those settings. Ideally, I'd like to see weak-to-strong or pooled-to-strong generalization to much harder tasks or multi-turn settings. The cleanest experiment to get me to flip would be this sort of far-out-OOD generalization of monitors for the strongest models that you have. Qwen3-32B is near-saturated on MATH/GSM8K, and maybe not there for some TerminalBench or equivalent. Does Qwen3-32B perform better than SC or RLM baselines if you introduce self-Agent? Another acceptable demonstration would be to show training efficiency using these monitors or some other experiment-backed argument for why these probes might be useful during training to make models better at catching silly mistakes in a more data-efficient way. If there is a single reason I can point to to justify rejecting this paper, it would be this problem.

(3) More probing of the rewrite modules: more ink has been spilled (especially in the Appendix) on how to train the monitor than how to tune the rewriting module. This seems like an oversight, given that many other works have broached the questions of inference-time harnesses for self-optimization (see Jason Weston's work to this effect). I wonder if the re-writers generalize cleanly across tasks, or if there is some hill-climbing to be done with JEPA or another evolutionary algorithm to evolve a stronger prompt for this setup. Not necessary, but would be good. As of now, it reads as if the authors spent more time thinking about the monitors.

(4) A few more monitor-based analyses would be nice. The authors do include many comparisons -- thank you for these, especially for comparison of different inputs -- but I would still like to see how a lightweight LLM monitor does (possibly with tuning) and some other more particular items that I'll enumerate below.

Overall, I lean towards a weak rejection. While the paper is well-written and produces an interesting kernel for further exploration, I would like to see a more careful treatment of related work (ideally w. stronger baseline comparisons), a much more extensive analysis of OOD generalization, weak-to-strong tasks for monitors, more diverse model types and more attention to the rewrite component of Self-Agent. In lieu of these, I would like to see a more grounded and concrete motivation for where Self-Agent would expect to be applied -- in production at inference-time for online monitoring, during training for more data-efficient RL, or somewhere else -- that concretely justifies why the current reported gains on mostly-saturated tasks by models represent a meaningful conceptual or applied contribution for working with reasoning models.

Questions for authors

(1) How does this work relate to studies of RLMs, linear probes used in other settings, uncertainty-based self-correction or other bootstrapped self-optimization strategies, especially those from 2026 and 2025? What empirical results can you demonstrate about the novelty or superiority of your method relative to those baselines?

(2) The results presented are on models which have already saturated their benchmarks, except for AIME. What do harder benchmarks look like with this analysis? How well does the monitor training process generalize across new tasks (no need to necessarily re-do the inference and accuracy loops, training for the monitors alone is fine, even offline using something like the OpenThoughts dataset)? More importantly, how do monitors generalize from easy-to-verify to hard-to-verify tasks if trained with a broader mixture of supervised training data? Can you provide statistics other than AUROC/AUPRC (which are threshold-blind, and thus don't tell me much about your chosen γ\gamma threshold) to describe how well these monitors learn? If not, what is the motivation for using monitors for tasks models have already trained on? Some examples of OOD properties I'd want to look at:

  • Multi-turn coding settings, where the format of the transcript is materially different.
  • Objective tasks that involve natural language, like MMLU or GPQA, or instruction following, like IFEval, where the grader is independent of stating an answer but checks against compliance to questions. BFCL would be good here. - Multi-agent benchmarks, since you reference these as a third leg for your definition of "reliability" but don't really clarify the relationship. - How well do monitors work off-policy? Preliminary Qwen-32B --> 8B experiment suggests not very well, so the monitors are only good at the on-policy activation comparison. That's ok, but I'd like to see more!

(3) What do results look like on models outside of the Qwen family?

(4) When choosing your method for drawing binary labels for training your classifier, what threshold did you choose? Did you try training a regression? How well did it work?

(5) What is the purpose for using monitors on tasks that models have already saturated on? It seems like, based on the hardness-stratification analysis, it is the easiest problems where most of the gains in absolute terms are shown. These are the first to go! What is the planned utility/finding here?

(4) How did you calibrate the rewrite module? Did you? How do other re-writing strategies compare, and why did you settle on the one which you chose?

(5) Some more monitor-based analyses:

  • Did you calibrate against human or strong-LLM annotations? Would be good to see these, esp. for hyperparams like choice of threshold. This may apply for γ\gamma and for the risk-thresholding based on resamples from a prefix.
  • Did you try other layers, including/excluding attention head positions, training per-prefix or per-length monitors, etc..?
  • Can you share more statistics on the monitors? Precision, recall, stratification and (LLM) annotation of where failures exist?
  • Can you residualize predictive accuracy against shallow predictors like length, bag-of-words heuristics that measure uncertainty, or other measures? Can you show the residual predictive strength of the activation-based classifiers w/r/t entropy?

(6) Please include CIs on results in Tables 1 and 2. Many of the results are single-digit percentage point improvements, even tenth-of-a-digit. Would be good to see both Wilcoxon and Mann-Whitney-based t-tests to assert significance, especially Wilcoxon to get a more comparative read.

(7) Minor nits: Figures 1 and 2 are not especially helpful for me personally, the self-consistency baseline could be more clearly explained in a demarcated setting, and limitations should be expanded in camera-ready if possible. I understand space is a constraint.

Reviewer_XUfKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Self-Agent, an online self-correction framework for frozen LLMs that integrates hidden-state-based risk monitoring, context-isolated error diagnosis, and conservative repair commitment. A lightweight reflection monitor is trained using synthetic prefix-level supervision derived from sampled continuations and outcome-level feedback. Experiments across Qwen3 models of different scales on mathematical reasoning and code generation demonstrate promising improvements with bounded inference overhead.

Critical strengths
  1. Relevant problem formulation. The paper addresses the limitations of existing self-correction approaches by introducing online monitoring and selective intervention at intermediate reasoning stages.

  2. Modular and lightweight design. The separation of risk estimation, context-isolated diagnosis, and selective repair is intuitive and avoids updating the underlying language model.

  3. Encouraging empirical evaluation. Experiments cover multiple model scales and task domains. The analysis of rescue/harm rates, inference costs, and component ablations provides useful insights into the proposed framework.

Critical weaknesses

W1. Limited evidence for local error detection. The monitor predicts continuation reliability rather than explicitly identifying local reasoning errors. Low continuation probability may reflect problem difficulty or model uncertainty rather than an actual error in the current prefix. The paper lacks controlled experiments to distinguish these factors.

W2. Insufficiently strong baselines. The budget-matched Self-Consistency baseline relies on random routing, which does not adequately control for adaptive test-time compute allocation. Comparisons with risk-guided Self-Consistency and risk-guided regeneration are needed to isolate the contribution of local diagnosis and repair.

W3. Insufficient details on training and evaluation protocols. Key aspects of continuation sampling, risk-label construction, and intervention-threshold selection are not sufficiently specified. In particular, the separation between monitor training, threshold calibration, and final evaluation should be clarified to establish reproducibility and rule out potential evaluation bias.

Questions for authors

Q1. Cross-model generalization. How well does the learned monitor generalize across models with different hidden-state representations? In particular, can the authors clarify the extent to which the reported 8B-to-14B transfer depends on re-encoding trajectories using the 8B model?

Q2. Statistical reliability. Given the small size of AIME 2025 (30 problems), the reported improvements correspond to only one or two additional correct answers. Do the authors have results across multiple independent generation seeds and paired statistical significance tests to establish the robustness of these gains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.