acceptodds
Under review as a conference paper at ICLR 2027

LLM-as-its-judge: A compute paradigm for self-aware and self-directing reasoning machines

Abstract

Large language models (LLMs) can perform multi-step reasoning to divide complex problems into a sequence of conquerable parts. These reasoning steps empowered LLMs for mathematical, logical, and algorithmic feats beyond one-step generation. Yet, with token generation being strictly autoregressive, early mistakes compound and subtle missteps can derail the entire reasoning trajectory. We demonstrate that LLMs can sense when these reasoning trajectories approach a tipping point to go off road. Then, by injecting specialized control tokens, LLMs can learn to push their own reasoning trajectories back on track. To bring “self-awareness” to LLM inference, we introduce *reflexive reasoning*, a novel reasoning paradigm complementary to current reasoning approaches. Across multiple standard benchmarks, reflexive reasoning improves the task performance of LLMs spanning a range of architectures and parameter counts, with no modification of LLM weights and no added inference-time compute. Through self-reasoning, we can also enable *reflexive tuning*, a fine-tuning framework to enforce tight policy and persona adherence at inference time, with fixed model weights. By repurposing the model’s internal representations to serve as its own observer and overseer, our approach delivers robust-by-design inference for complex, high-risk reasoning applications, with minimal computational overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reasoning, control, steering, LLMs, transformers, robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yrkgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new reflexive reasoning paradigm that leverages models’ internal representations through learned sensor and control tokens that propagate through the model’s forward pass. Experiments on reasoning and non-reasoning tasks using open-source models support the paper’s insights and demonstrate the effectiveness of the proposed paradigm.

Critical strengths
  • The results showing that sensor tokens can predict reasoning failures are interesting.

  • The evaluation covers both reasoning and non-reasoning tasks.

Critical weaknesses
  • The paper lacks ablation studies to isolate the contributions of the sensor and controller, and to compare performance with and without controller fine-tuning.

  • The novelty needs further justification. It is unclear how the proposed “introspective, closed-loop system” differs substantially from existing probing and continuous-token steering approaches. Such as: [1] Activation Steering with a Feedback Controller,ICLR 2026 [2] Programming Refusal with Conditional Activation Steering(CAST),ICLR 2025

  • The claim of “no added inference-time compute” is insufficiently supported. Sensor evaluation and control-token generation introduce additional computation.

Questions for authors

NA

4 comments
AuthorsOpenReview ↗

Author Rebuttal 1/2

We thank the reviewer for their thoughtful comments, and appreciate that they found our method interesting across a breadth of reasoning and non-reasoning tasks.

W1. We acknowledge that it may be beneficial to include additional controls and ablations. As such, we present new control experiments on random, periodic, fixed-rate, and unconditional control-token insertion. We find that our adaptive, sensor-driven insertion approach outperforms these methods comfortably, avoiding damaging insertions and only inserting control tokens at points that are helpful for reasoning recovery.

Qwen3-0.6B control token insertion variants on GSM8K test set

Intervention PolicyTrigger MechanismInvocation Rate (%)GSM8K pass@1 (%)Mean Added Tokens
CoT (No Control)None0.0%30.0%0.0
Random PlacementUniform random boundary73.3%31.7%2.2
Periodic InsertionEvery 30 tokens100.0%33.3%3.0
Fixed Safe BoundaryFirst natural boundary100.0%33.1%3.0
Unconditional InsertionInsertion at every boundary100.0%28.2%8.5
Reflexive ReasoningTrained sensor token68.8%40.1%4.2

We also report new results comparing SFT-only and SFT+GRPO sensor/control token training. We find that the additional GRPO step is essential for learning a robust and generalizable control token insertion policy. SFT serves as a useful warmup but is not sufficient by itself. We will add these results to the Appendix.

Qwen3-0.6B GSM8K test set pass@1 accuracy, SFT vs GRPO

Training stageAccuracy
SFT warm-start31.4%
SFT + GRPO40.1%

W2. We thank the reviewer for pointing out these related recent works. While the general idea of closed-loop control of LLM generation is not by itself novel, we believe that our novel contribution lies in how to implement this idea efficiently: using the frozen transformer itself to perform both learned state sensing and recurrent, on-the-fly construction of context-dependent interventions. In particular, prior activation steering techniques (such as CAST and feedback controller steering) operate by patching intermediate hidden states with additive offsets at specific layers. In direct contrast, our approach operates strictly within sequence space, taking advantage of the recurrent, autoregressive nature of the transformer and the full capacity of its representation space to monitor its own reasoning trajectory and design control adjustments on-the-fly. This last piece is important: each successive control token is formed by the LLM, conditioned on the intermediate hidden state of the preceding control token and the entire sequence prefix, allowing the intervention to adapt to the evolving generation context rather than applying a static vector offset. CAST by contrast relies on clamping activations along fixed, pre-extracted semantic directions, making it naive to the model’s own preceding generations. This is a fundamental difference in philosophy. Traditional activation steering approaches could be thought of as operating in static activation space, whereas our approach effectively leverages the essential temporal aspect of the transformer. We are convinced that respecting the temporal nature of the LLM is necessary to enable robust control, and indeed this approach has allowed us to explore more difficult reasoning tasks such as math, formal logic, and code which were not explored in CAST or the feedback controller paper (tested on simple refusal tasks, not complex reasoning).

Further, we show that our learned continuous sensor token captures trajectory-level derailment far more effectively than traditional linear probes; as reported in Section 4.1 and Figure 2A, the sensor achieves an AUC of 0.807 on GSM8K failure prediction, significantly outperforming both a trained linear probe on preceding hidden states (AUC 0.695). Taken together, we believe that our approach opens the door to harness the LLM’s representation space and recurrent nature to build more complex and programmable reasoning machines for monitoring, control, and capability enhancement. We will add these comparisons to the updated Related Work section to better highlight the novelty of our approach.

AuthorsOpenReview ↗

Author Rebuttal 2/2

W3. We agree that how we phrased our claim of “no added inference-time compute” was too drastic. We meant to illustrate that our method substantially enhances reasoning performance without expensive inference-time rollouts, re-generations, tree/beam search, or multi-turn self-correction. Indeed, our core contribution is to show that we can repurpose the LLM’s forward pass to manage its own targeted monitoring and intervention. To enable this closed-loop introspection, we add very lightweight linear classifiers and gates, corresponding to only 0.0125% additional parameters on top of Gemma-3 4B. This is a negligible increase of 0.038% extra FLOPs compared to standard discrete token generation. Still, we understand that detailed efficiency profiles of our method would be quite useful. Therefore we will include a complete efficiency panel on resource usage and latency for our method compared to soft prompting and the standard chain-of-thought baseline:

ModelMethodExtra Params (%)FLOPs/Token OverheadPeak VRAM (GB)Mean Output TokensTotal Processed TokensLatency p50 (s)Latency p95 (s)
Qwen3-0.6BGreedy CoT0.000%+0.00%1.85377.8498.22.453.82
Soft Prompt (20 tokens)+0.003%+1.92%1.86328.4468.82.223.51
Reflexive Reasoning (Ours)+0.019%+0.04%1.86233.2361.61.682.74
Qwen3-1.7BGreedy CoT0.000%+0.00%4.12383.6504.13.885.92
Soft Prompt (20 tokens)+0.002%+1.92%4.14351.2491.73.615.48
Reflexive Reasoning (Ours)+0.015%+0.04%4.15183.9312.42.153.42
Gemma-3-4B-ITGreedy CoT0.000%+0.00%9.42203.2323.53.124.85
Soft Prompt (20 tokens)+0.001%+1.58%9.45198.9339.23.104.81
Reflexive Reasoning (Ours)+0.0125%+0.038%9.48248.7376.93.455.21

Finally, we will re-phrase the sentence stating “no added inference-time compute” to “Across multiple standard benchmarks, reflexive reasoning improves task performance across model architectures and sizes while keeping backbone weights frozen and operating within a single generation trajectory, without separate judge-model evaluations.”

Thanks for the response. I'd like to keep my score.

Could you explain why you would like to keep your score, considering that we directly addressed each weakness in your original review?

We added full "ablation studies to isolate the contributions of the sensor and controller, and to compare performance with and without controller fine-tuning" (table of comparisons with different sensor trigger mechanisms, a direct comparison of SFT only versus SFT+GRPO). We compared the novelty of our approach, in detail, with Activation Steering with a Feedback Controller, ICLR 2026 and Programming Refusal with Conditional Activation Steering (CAST), ICLR 2025. Finally, we add a full efficiency profiling table to more fully support our claim of “no added inference-time compute”, and we soften the language to avoid over-claiming on the efficiency front.

We are happy to further clarify any additional points. Thank you for your time.

Reviewer_YzgvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reflexive Reasoning, a framework that enables language models to monitor and steer their own reasoning trajectories during inference without modifying the underlying model weights. The method introduces a learnable sensor token to predict potential reasoning failures from intermediate hidden states and a set of recurrently constructed control tokens to redirect subsequent generation when intervention is deemed necessary. The sensor is trained using outcome-based supervision, while the controller is optimized through supervised fine-tuning and GRPO. The authors evaluate the approach on mathematical reasoning (GSM8K), logical reasoning (PrOntoQA), and code reasoning (CRUXEval) using Qwen3 and Gemma-3 models. They also investigate instruction following and behavioral constraints. The reported results show improvements in final-answer accuracy over standard chain-of-thought and soft prompting in several settings, together with reductions in generated token counts.

Critical strengths

S1. An interesting approach to online reasoning control.

The paper explores a promising direction by integrating failure prediction and corrective interventions within a single reasoning trajectory, rather than relying on external judges or repeated generations. The use of learned continuous tokens with frozen model weights provides a potentially useful mechanism for dynamically steering model generation.

S2. Encouraging empirical results across multiple reasoning tasks.

The reported results show improvements over standard CoT on mathematical, logical, and code reasoning benchmarks across several model architectures. The separate evaluations of sensor prediction and controller steering also provide preliminary evidence that the proposed components can influence reasoning outcomes.

S3. Useful analysis of the proposed mechanisms.

Beyond the end-to-end benchmark results, the paper provides additional analyses of sensor prediction, controller steering capacity, and the internal representations of the learned tokens. These experiments offer useful insights into how the proposed mechanism operates, although further validation is needed to establish the individual contributions of its components.

Critical weaknesses

W1. Insufficient validation of sensor supervision and critical intervention point identification.

The sensor is trained using trajectory-level correctness labels derived from final answers. However, predicting whether a reasoning trajectory will eventually fail is different from identifying intermediate points where corrective intervention is beneficial. A sensor may learn to predict eventual failure based on task difficulty or trajectory characteristics without accurately identifying actionable intervention points.

Although the paper reports a ROC-AUC of 0.807 for failure prediction, this metric does not directly measure intervention utility. Furthermore, the reported 75.7% recovery rate relies on oracle intervention positions, while the end-to-end evaluation does not provide a detailed analysis of the benefits and potential harms of actual sensor-triggered interventions. Therefore, additional evidence is needed to establish whether the sensor reliably identifies when intervention is beneficial, rather than merely predicting eventual failure.

W2. Insufficient ablations to establish the contribution of dynamic intervention.

Section 5.1 demonstrates that the complete framework improves performance over standard CoT and several alternative methods. However, these comparisons do not isolate the contribution of the proposed sensor-based intervention policy. In particular, the paper does not compare sensor-triggered interventions with fixed-position or randomly triggered control tokens under matched intervention budgets. Such comparisons are important for determining whether the sensor's adaptive selection of intervention points provides additional benefits beyond the learned control tokens themselves.

Furthermore, while the paper separately evaluates sensor prediction and controller steering, it remains unclear how much each component and training stage contributes to the final performance. Ablations of the sensor-triggering mechanism and SFT-only versus SFT+GRPO training would help clarify which elements of the framework are essential.

W3. Insufficient differentiation from related methods and limited evidence of cross-task generalization.

The proposed framework shares similarities with existing approaches such as soft prompt tuning, activation steering, and representation engineering. Although the paper discusses several differences from prior work, it remains unclear whether the proposed dynamic intervention mechanism offers substantial advantages over closely related adaptive steering methods. More direct comparisons with these approaches would help better establish the methodological contribution of the proposed framework.

Furthermore, the sensor and controller are trained using task-specific reasoning trajectories and outcome supervision. While the method demonstrates improvements across multiple benchmarks and model architectures, the experiments do not establish whether a trained sensor-controller system can generalize to unseen tasks or reasoning domains without additional optimization. This limits the evidence supporting the broader generality and applicability of the proposed reasoning paradigm.

W4. Limited analysis of intervention sensitivity and stability.

The proposed framework relies on several design choices, including the sensor triggering threshold, the number of control tokens, and the controller training procedure. However, the paper provides limited analysis of how sensitive the final performance is to these choices. In particular, it remains unclear whether the reported improvements are stable across different intervention thresholds, control-token configurations, and random seeds. Additional sensitivity analyses would help establish the robustness and reliability of the proposed framework.

Questions for authors

Q1. Sensor supervision and evaluation.

How are trajectory-level correctness labels assigned to intermediate prefixes, and is any additional step-level or intervention-based supervision used? Is the reported ROC-AUC computed at the prefix level or trajectory level? Can the authors compare against a prompt-only failure predictor and evaluate correct versus incorrect rollouts of the same question to better distinguish trajectory-specific failure prediction from task difficulty?

Q2. Oracle intervention and component ablations.

How are the oracle critical points identified for the reported 75.7% recovery rate? Can the authors report recovery and regression rates under actual sensor-triggered interventions, together with fixed-position, random-position, and controller-only baselines under matched intervention budgets? How much of the final improvement comes from SFT versus GRPO?

Q3. Methodological novelty and cross-task generalization.

What are the key methodological differences between the proposed framework and existing adaptive activation steering approaches, particularly those involving closed-loop control? Can the authors provide more direct comparisons to demonstrate the advantages of the proposed intervention mechanism? Additionally, can a sensor-controller system trained on one task generalize to unseen tasks or reasoning domains without additional optimization?

Q4. Computational efficiency and robustness.

Can the authors report end-to-end inference latency and computational overhead, including sensor evaluations and control-token construction? How stable are the reported improvements across different intervention thresholds, control-token configurations, and random seeds?

3 comments
AuthorsOpenReview ↗

Author Rebuttal 1/3

We thank the reviewer for the helpful suggestions. We are pleased that the reviewer found our novel method to be a promising direction for online reasoning control, and that they note our strong empirical results across a wide array of benchmark tasks.

W1. We acknowledge that reasoning failure prediction versus intervention usefulness prediction serve different goals. While we primarily train the sensor tokens to predict reasoning failures, we also include a final calibration step after the initial sensor and control tokens have been trained to predict whether the control tokens will provide useful intervention. It is necessary to perform this step at the end of training, otherwise the sensor tokens will rely on poorly trained control tokens for their training signals. Therefore, training the sensor tokens to predict reasoning failure initially leads to much more stable training for the sensors and controls, which can then be bootstrapped to allow the sensors to calibrate towards their main objective of predicting useful intervention points. We will clarify these calibration details in the updated Methods section.

The goal of one of our experiments (Section 4) was to demonstrate the feasibility of re-using the forward pass of the LLM for introspective monitoring of reasoning failures in the first place (Section 4.1). This is a genuinely novel result, which we build on to implement the full reflexive reasoning approach. Next, we investigate the capacity of the control tokens to modify downstream text, independent of identifying the optimal insertion point (Section 4.2). With these two building blocks established, we combine them in Section 5 to demonstrate the combined power of introspective monitoring and steering of reasoning trajectories in our closed loop system. Therefore the 75.7% recovery oracle result is presented as an upper limit on the control token effectiveness, assuming that we have the perfect ability to identify the optimal places for intervention in the reasoning trajectory (this is not true in practice, as the sensor tokens naturally have some error associated with their predictions). We provide a more detailed breakdown of intervention recoveries and regressions here, and we will update the wording in Section 4 to motivate these results more clearly.

DatasetModelNBase Acc (%)Reflexive Acc (%)Net Gain (pp)Wrong->Right (Rec.)Right->Wrong (Reg.)Net RescuedInvocation Rate (%)
GSM8KQwen3-0.6B131930.0%40.1%+10.1%211113+9868.8%
GSM8KQwen3-1.7B131919.5%56.9%+37.4%52962+46796.3%
GSM8KGemma-3-4B-IT131975.1%77.2%+2.1%4125+1618.2%
PrOntoQAQwen3-0.6B50028.4%44.6%+16.2%185104+8198.4%
PrOntoQAGemma-3-4B-IT50042.6%53.2%+10.6%12370+53100.0%
CRUXEvalQwen3-0.6B30029.0%34.3%+5.3%4125+1674.3%
AuthorsOpenReview ↗

Author Rebuttal 2/3

W2. Motivated by this feedback, we provided more control experiments to isolate the effects of the sensor and control tokens, respectively. We ran additional experiments on random, periodic, fixed-rate, and unconditional control-token insertion. We find that our adaptive sensor regime provides substantial benefit over these rudimentary control token insertion strategies.

Qwen3-0.6B control token insertion variants on GSM8K test set

Intervention PolicyTrigger MechanismInvocation Rate (%)GSM8K pass@1 (%)Mean Added Tokens
CoT (No Control)None0.0%30.0%0.0
Random PlacementUniform random boundary73.3%31.7%2.2
Periodic InsertionEvery 30 tokens100.0%33.3%3.0
Fixed Safe BoundaryFirst natural boundary100.0%33.1%3.0
Unconditional InsertionInsertion at every boundary100.0%28.2%8.5
Reflexive ReasoningTrained sensor token68.8%40.1%4.2

Following this reviewer’s suggestion, we report new results comparing SFT-only and SFT+GRPO sensor/control token training. We find that the additional GRPO step is essential for learning a robust, generalizable control token insertion policy. SFT serves as a useful warmup but is insufficient alone; GRPO improves GSM8K test set performance of Qwen3-0.6B with reflexive reasoning by 8.7 percentage points. We will add these results to the Appendix.

Qwen3-0.6B GSM8K test set pass@1 accuracy, SFT vs GRPO

Training stageAccuracy
SFT warm-start31.4%
SFT + GRPO40.1%

W3. We appreciate the opportunity to conceptually differentiate our work more clearly from prior methods. Unlike standard soft prompt tuning, which prepends a fixed sequence of continuous tokens to the prompt before generation begins, reflexive reasoning is dynamic, mid-generation, and sparse. Static prefix prompts cannot adapt once reasoning starts to drift, whereas our sensor monitors trajectory vitals and injects control tokens only at critical inflection points where an impending error is detected. As demonstrated in Table 1, this dynamic intervention mechanism outperforms soft prompt tuning by an average of 12.1% in final answer accuracy across GSM8K, PrOntoQA, and CRUXEval and across all tested model architectures. Reflexive reasoning also differs from adaptive activation steering and representation engineering. Rather than perturbing internal layer activations via pre-computed additive vectors, our approach uses the model's native recurrent token sequence. Each successive control token is formed by the LLM, conditioned on the intermediate hidden state of the preceding control token and the entire sequence prefix, allowing the intervention to adapt to the evolving generation context rather than applying a static vector offset. Traditional activation steering approaches could be thought of as operating only within static activation space, ignoring the token sequence, whereas our approach effectively leverages the essential temporal aspect of the transformer. We believe respecting the temporal nature of the LLM is necessary to enable robust control, and this approach has allowed us to explore more difficult reasoning tasks such as math, formal logic, and code. We will add these motivations in the Introduction and Related Work sections to better delineate our novel contributions above existing soft prompting and activation engineering approaches. See also our response to reviewer “yrkg” for comparison to activation steering approaches CAST [1] and feedback controller approaches [2].

We appreciate the suggestion from this reviewer to show results on cross-task generalization. We include new results showing that our method trained on the training set of GSM8K, PrOntoQA, and CRUXEval demonstrates cross-benchmark improvement over standard chain-of-thought in almost all settings. Notably, we see the greatest lift from training on the larger GSM8K training set and applying these sensors and control tokens to PrOntoQA and CRUXEval, even narrowly beating the in-domain performance of reflexive reasoning on these benchmarks. Thus, we would expect that training our sensor and control token system on many general reasoning examples would lead to even larger gains. We will include these new generalization results in our updated Appendix.

Generalization accuracy of Qwen3-0.6B with reflexive reasoning trained on different benchmarks

Target: GSM8K (Math)Target: PrOntoQA (Logic)Target: CRUXEval (Code)
Frozen Baseline (No Control)30.0%28.4%29.0%
Trained on GSM8K (Math; n=5,500)40.1% (In-Domain)51.4%35.1%
Trained on PrOntoQA (Logic; n=500)27.2%44.6% (In-Domain)33.8%
Trained on CRUXEval (Code; n=500)31.6%41.2%34.3% (In-Domain)

[1] Programming Refusal with Conditional Activation Steering, Lee et al. 2024 [2] Activation Steering with a Feedback Controller, Nguyen et al. 2025

AuthorsOpenReview ↗

Author Rebuttal 3/3

W4. We appreciate this suggestion from the reviewer. To address these concerns, we ran additional new analyses on multiple thresholds, control-token configurations, and random seeds. We find that our validation set selection procedure for threshold and control-token configuration is still the optimal way to choose these settings. We find that our method is quite stable, with a standard deviation of only 0.56% on the GSM8K test set for Qwen3-0.6B across 5 different random seeds. We will add these additional sensitivity analyses to the updated Appendix.

GSM8K test set accuracy results for Qwen3-0.6B at different sensor token thresholds

Threshold (tau)Invocation Rate (%)pass@1 (%)
tau = 0.2089.6%38.2%
tau = 0.3079.5%37.4%
tau = 0.39 (validation set)68.8%40.1%
tau = 0.6042.9%36.8%
tau = 0.7029.6%35.3%
tau = 0.951.6%30.1%

GSM8K test set accuracy results for Qwen3-0.6B with different numbers of control tokens

Control Token Count (K)pass@1 (%)
K = 127.6%
K = 238.7%
K = 3 (validation set)40.1%
K = 438.4%
K = 530.3%

Q1. We perform an additional calibration step after both the sensor and control tokens are initially trained, in order to ensure stable training. Details are given in response to W1. ROC-AUC is reported for each prefix, making it a necessarily conservative estimate, as early sensor placement will naturally be more error-prone. Indeed, classifying just the prompt gives us an error prediction AUC of only 0.67 on the GSM8K test set with Qwen3-0.6B rollouts, compared with full-trajectory error prediction AUC of 0.81. As well, we evaluate the sensor’s ability to predict correct/incorrect reasoning trajectories on matched rollouts of the same prompt from the GSM8K test set. We find that we are able to predict the correct/incorrect reasoning trajectory from these matched rollouts with an AUC of 0.78, showing that our sensor tokens do indeed predict reasoning failures, not just prompt difficulty. We will add these useful results to Section 4.1, and thank the reviewer for suggesting these analyses.

Q2. The oracle points are determined by generating a full trajectory rollout, then testing the control token insertion at various points along those rollouts. We then keep only the insertion points that do not lead to regression on questions that the model would have gotten correct without intervention. This provides an upper bound on our reflexive reasoning approach with a hypothetical perfectly accurate sensor; beyond that, the control tokens do not have enough capacity to influence the model towards a correct answer. We report additional control experiments in our response to W2.

Q3. We provide more methodological comparisons in our response to W3. In short, our reflexive reasoning approach does generalize across benchmarks, although this generalization performance is largely dependent on the training set sample size (the full table is provided in response to W3).

Q4. We report sensitivity analyses in our response to W4. We also completed a more fulsome efficiency evaluation, which we reproduce below, showing that reflexive reasoning adds little if any inference latency/efficiency penalty (GSM8K):

ModelMethodExtra Params (%)FLOPs/Token OverheadPeak VRAM (GB)Mean Output TokensTotal Processed TokensLatency p50 (s)Latency p95 (s)
Qwen3-0.6BGreedy CoT0.000%+0.00%1.85377.8498.22.453.82
Soft Prompt (20 tokens)+0.003%+1.92%1.86328.4468.82.223.51
Reflexive Reasoning (Ours)+0.019%+0.04%1.86233.2361.61.682.74
Qwen3-1.7BGreedy CoT0.000%+0.00%4.12383.6504.13.885.92
Soft Prompt (20 tokens)+0.002%+1.92%4.14351.2491.73.615.48
Reflexive Reasoning (Ours)+0.015%+0.04%4.15183.9312.42.153.42
Gemma-3-4B-ITGreedy CoT0.000%+0.00%9.42203.2323.53.124.85
Soft Prompt (20 tokens)+0.001%+1.58%9.45198.9339.23.104.81
Reflexive Reasoning (Ours)+0.0125%+0.038%9.48248.7376.93.455.21

In the updated paper, we will also discuss the potential of steering and model control for potential misuse, along with monitoring strategies to mitigate these issues.

Reviewer_NYbwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes reflexive reasoning, a framework for controlling frozen language models during autoregressive generation. It introduces a learned sensor token to detect potential reasoning errors at textual boundaries and conditionally inserts control tokens to redirect subsequent generation. The sensor, controller, and gating parameters are trained through supervised repair continuations and outcome-based policy optimization without updating backbone weights. Experiments across Qwen3 and Gemma-3 models demonstrate improvements over standard chain-of-thought and soft prompting on reasoning benchmarks, with additional comparisons against LoRA and self-consistency. The main contribution is conditional mid-generation control through continuous tokens, although claims regarding introspective self-awareness and negligible inference overhead remain insufficiently supported.

Critical strengths
  1. An interesting conditional control mechanism for frozen LLMs. The sensor–controller decomposition (Section 3.2) enables intervention at intermediate points rather than only through an initial soft prompt. Because the backbone remains frozen, the approach explores a meaningful adaptation regime between static prompt tuning and full weight updates.

  2. Direct evidence that intermediate representations predict failures. On held-out GSM8K problems, the sensor reaches an AUC of 0.807, compared with 0.557 for next-token entropy and 0.695 for a trained hidden-state probe (Section 4.1 and Figure 2). This is a substantive empirical observation supporting the usefulness of learned readout tokens, independently of whether the process constitutes “self-awareness.”

  3. A separate controllability test, including negative outcomes. Section 4.2 and Appendix A.2 evaluate what injected continuous tokens can change before combining them with a learned trigger. The oracle-placement experiment recovers 785/1,037 initially incorrect GSM8K cases but also causes 53/282 initially correct trajectories to regress (Table 5 in the appendix). Reporting both rescue and harm is important because it demonstrates that correct intervention timing, not merely controller strength, is necessary.

  4. End-to-end tests across several task types and backbones. Table 1 evaluates mathematics, symbolic logic, and code-related reasoning with Qwen3-0.6B, Qwen3-1.7B, and Gemma-3-4B-IT. Improvements over ordinary CoT are visible in a number of settings, notably Qwen3-1.7B GSM8K (19.5% to 56.9%). The breadth makes this more than a toy proof of concept.

Critical weaknesses
  1. The no-added-inference-compute claim is unsupported and seemingly inconsistent with the algorithm. The Abstract and Section 1 assert “no added inference-time compute.” Yet Section 3.2 requires sensor-token forward computation at candidate boundaries, an auxiliary classifier, up to K=3 inserted control tokens, recurrent token construction, and gating computations. Some generated text responses are shorter (Table 1), but others are longer; neither output-token count nor frozen weights equals zero incremental FLOPs, KV-cache use, or latency. A concrete end-to-end profiling table is essential. The defensible statement may be that the method avoids separate full-model judges or additional sampled trajectories—not that it adds no computation.

  2. Failure prediction should not be conflated with detection of an imminent correctable mistake. Section 3.3 trains the sensor using labels from final-answer correctness. Consequently, a prefix classifier can exploit broad task difficulty, early problem content, or prefix length without localizing a specific impending deviation. Section 4.1 demonstrates predictive AUC, but this does not show that the sensor identifies the precise transition at which an intervention becomes beneficial. This matters because the paper's narrative invokes introspection, anticipation, and a closed-loop internal governor. A held-out, matched-trigger-frequency experiment correlating predicted scores with causal repair benefit is needed to support that stronger interpretation.

  3. Key baselines are not fully controlled for intervention budget and placement. Table 1 compares against a 20-token prefix soft prompt, whereas the proposed method uses conditional mid-sequence insertion with a trained sensor, multiple control steps, and outcome-based refinement. These methods differ in training objective, number of effective interventions, and position as well as token representation. Appendix Table 6 includes LoRA and self-consistency, which is useful, but neither isolates the added value of adaptive triggering over random, periodic, fixed-rate, or unconditional control-token insertion. Without these controls, it is difficult to know which component drives the performance gains.

  4. The inference efficiency benefit is not uniform across tasks. Table 1 emphasizes lower generated-token counts in many settings, but Gemma-3-4B-IT uses 248.7 versus 203.2 response tokens on GSM8K, and the comparison does not include nontext control-token computation. Section 4.2 also documents an 18.8% oracle-control regression rate on initially correct answers. It is therefore premature to characterize the design as generally compute-free or robust-by-design. Latency distributions, invocation rates, false-trigger costs, and wrong-to-right/right-to-wrong outcomes should be shown for the final end-to-end policy.

Questions for authors
  1. What are the measured p50/p95 end-to-end latency, GPU FLOPs/time, peak memory, and total processed tokens—including sensor, controller, and auxiliary-network operations—compared with standard CoT and a same-compute prompt-tuning baseline?

  2. For matched intervention count and position distribution, how do random triggers, periodic triggers, a confidence/entropy trigger, unconditional controller insertion, and the learned sensor compare? Does sensor score predict the difference between controlled and uncontrolled success?

4 comments
AuthorsOpenReview ↗

Author Rebuttal 1/2

We thank the reviewer for their detailed and thoughtful comments on our paper. We appreciate that the reviewer found our method to be an interesting and meaningful adaptation regime between prompt tuning and full weight updates. As well, we appreciate that the reviewer noted the substantive empirical results and breadth of our experiments and benchmarks.

W1. Our sensor-control closed loop only adds 0.0125% parameters and 0.038% extra FLOPs (Gemma-3 4B). Nevertheless, we agree with the reviewer that “no added inference-time compute” might be too strong of a claim. We meant to illustrate that our method substantially enhances reasoning performance without expensive inference-time rollouts, re-generations, tree/beam search, or multi-turn self-correction. Indeed, our core contribution is to show that we can repurpose the LLM’s forward pass to manage its own targeted monitoring and intervention. To enable this closed-loop introspection, we add very lightweight linear classifiers and gates, corresponding to only 0.0125% additional parameters on top of Gemma-3 4B. This is a negligible increase of 0.038% extra FLOPs compared to standard discrete token generation. In response to the reviewer’s suggestion, we provide a complete profiling table of resource usage and latency for our method compared to soft prompting and the standard chain-of-thought baseline (GSM8K):

ModelMethodExtra Params (%)FLOPs/Token OverheadPeak VRAM (GB)Mean Output TokensTotal Processed TokensLatency p50 (s)Latency p95 (s)
Qwen3-0.6BGreedy CoT0.000%+0.00%1.85377.8498.22.453.82
Soft Prompt (20 tokens)+0.003%+1.92%1.86328.4468.82.223.51
Reflexive Reasoning (Ours)+0.019%+0.04%1.86233.2361.61.682.74
Qwen3-1.7BGreedy CoT0.000%+0.00%4.12383.6504.13.885.92
Soft Prompt (20 tokens)+0.002%+1.92%4.14351.2491.73.615.48
Reflexive Reasoning (Ours)+0.015%+0.04%4.15183.9312.42.153.42
Gemma-3-4B-ITGreedy CoT0.000%+0.00%9.42203.2323.53.124.85
Soft Prompt (20 tokens)+0.001%+1.58%9.45198.9339.23.104.81
Reflexive Reasoning (Ours)+0.0125%+0.038%9.48248.7376.93.455.21

We will also re-phrase the sentence stating “no added inference-time compute” to “Across multiple standard benchmarks, reflexive reasoning improves task performance across model architectures and sizes while keeping backbone weights frozen and operating within a single generation trajectory, without separate judge-model evaluations.”

W2. We appreciate the reviewer for pointing out this detail. We do not claim to identify the singular, precise point at which intervention becomes beneficial. Indeed, there are likely many such points along a reasoning trajectory where intervention is necessary and helpful. As well, the entire purpose of the sensor token is to access and use the model’s internal states to “look ahead” and determine if the current reasoning trajectory is heading down an incorrect path. We should also clarify that we do in fact perform a final sensor tuning step once the sensor and control tokens are trained independently to reduce regressions introduced by the control tokens, and to detect “correctable mistakes”. It is necessary to introduce this step once the sensor and control tokens have been trained, otherwise the control tokens will be introduced at moving targets, and will not be able to learn effectively. We will clarify this calibration step and its rationale in the updated Methods section. To demonstrate that these fully calibrated sensor tokens properly track these “correctable mistakes”, we correlate the sensor token predictions with the repair benefit introduced by the triggered control tokens, as suggested by the reviewer. For Qwen3-0.6B, we see a Pearson correlation of r=0.412r = 0.412 between these sensor predictions and binary trajectory recovery outcome, showing that our sensor tokens do indeed track helpful recoveries, on average. We will add these results to Section 5.1.

AuthorsOpenReview ↗

Author Rebuttal 2/2

W3. Our results in the main text show that our adaptive mid-sequence approach outperforms soft prompts on every benchmark from math to coding, often by as much as 20.9 percentage points. We believe that a key contribution of our method is that (adaptive) mid-sequence placement of control tokens can provide substantial benefit for reasoning performance beyond the prefix placement of soft prompts. Still, we understand the reviewer’s point that added control experiments would isolate the contribution of the sensor tokens from the control tokens. Therefore, in accordance with the reviewer’s helpful suggestion, we report new results on random, periodic, fixed-rate, and unconditional control-token insertion. We find that our adaptive, sensor-driven insertion approach outperforms these methods comfortably, avoiding damaging insertions and only inserting control tokens at points that are helpful for reasoning recovery.

Qwen3-0.6B control token insertion variants on GSM8K test set

Intervention PolicyTrigger MechanismInvocation Rate (%)GSM8K pass@1 (%)Mean Added Tokens
CoT (No Control)None0.0%30.0%0.0
Random PlacementUniform random boundary73.3%31.7%2.2
Periodic InsertionEvery 30 tokens100.0%33.3%3.0
Fixed Safe BoundaryFirst natural boundary100.0%33.1%3.0
Unconditional InsertionInsertion at every boundary100.0%28.2%8.5
Reflexive ReasoningTrained sensor token68.8%40.1%4.2

W4. While our reflexive reasoning approach does not lead to lower token usage in each and every instance across all settings, we do empirically see 54% fewer tokens used on average (225 average tokens for reflexive reasoning across models/benchmarks, compared to 489 average tokens for standard chain-of-thought). We should also clarify that the non-text sensor/control tokens are a very very small part of the overall token count; these sensors are triggered a median of 2 times for each sample across benchmarks, leading to a median extra token count of 8, less than 4% of the total number of tokens used in these reasoning trajectories. In addition to this token efficiency, the additional linear machinery required to implement reflexive reasoning is negligible compared to the transformer forward pass operations, as detailed in our response to W1. Still, we understand that the current text could have more evidence for these claims. Therefore we will add a new efficiency table to the Appendix detailing reflexive reasoning invocation rates, false-trigger costs, and wrong-to-right/right-to-wrong outcomes:

DatasetModelNBase Acc (%)Reflexive Acc (%)Net Gain (pp)Wrong->Right (Rec.)Right->Wrong (Reg.)Net RescuedInvocation Rate (%)
GSM8KQwen3-0.6B131930.0%40.1%+10.1%211113+9868.8%
GSM8KQwen3-1.7B131919.5%56.9%+37.4%52962+46796.3%
GSM8KGemma-3-4B-IT131975.1%77.2%+2.1%4125+1618.2%
PrOntoQAQwen3-0.6B50028.4%44.6%+16.2%185104+8198.4%
PrOntoQAGemma-3-4B-IT50042.6%53.2%+10.6%12370+53100.0%
CRUXEvalQwen3-0.6B30029.0%34.3%+5.3%4125+1674.3%

Q1. See the new efficiency table in response to W1.

Q2. Yes, sensor score does predict the difference between controlled and uncontrolled success. See our detailed control experiments in responses to W2 and W3.

Thanks for your response. But the problems still exists. I would keep my scores.

Could the reviewer clarify "which problems still exists"? We have addressed every one of your weaknesses and questions. In fact, we directly include a new table for Q1 containing each requested metric:

What are the measured p50/p95 end-to-end latency, GPU FLOPs/time, peak memory, and total processed tokens—including sensor, controller, and auxiliary-network operations—compared with standard CoT and a same-compute prompt-tuning baseline?

This table reveals that our method provides more robust reasoning capability at the virtually the same compute budget as current state of the art methods like CoT and soft prompting.

As well, we address Q2 directly with new control experiments:

For matched intervention count and position distribution, how do random triggers, periodic triggers, a confidence/entropy trigger, unconditional controller insertion, and the learned sensor compare? Does sensor score predict the difference between controlled and uncontrolled success?

Each of random triggers, periodic triggers, a confidence/entropy trigger, and unconditional controller insertion perform significantly worse than our learned sensor. Our sensor score does indeed predict the difference between controlled and uncontrolled success.

Given that there are no remaining outstanding concerns, we would appreciate if the reviewer re-considered their initial score. We would be happy to provide any additional clarifications on our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General author rebuttal/summary

We thank the reviewers for their helpful comments and suggestions. We have addressed each weakness and question from all reviewers in dedicated responses; we provide a summary here, and we have now uploaded a revised paper based on this feedback, with changes and additions highlighted in green.


Shared reviewer points

Inference-time compute claims and efficiency profiling (NYbw W1, yrkg W3, Yzgv Q4): We have softened the "no added inference-time compute" claim in the Abstract and the Introduction. We clarify that reflexive reasoning operates within a single generation trajectory with frozen backbone weights and no separate judge-model evaluations. In addition, we added a complete GSM8K profiling table (extra parameters, FLOPs/token overhead, peak VRAM, processed tokens, and p50/p95 latency) in Appendix A.6 (Table 7) showing that our approach provides more robust reasoning capability at nearly the same compute budget as standard chain-of-thought.

Sensor token calibration (NYbw W2, Yzgv W1): We clarified in Section 3.3 that we perform a sensor calibration step after the initial sensor and controller training to home in on correctable mistakes and avoid regressions introduced by the control tokens. We first train the sensors to detect reasoning failures, and then tune them to only fire when they estimate that control token insertion would be helpful.

Oracle recoveries and end-to-end recoveries and regressions (NYbw W4, Yzgv W1/Q2): We updated Section 4 and Section 4.2 to explain how oracle insertion points are identified. We also added end-to-end recovery, regression, net rescued counts, and invocation rates across benchmarks in Appendix A.6 (Table 8).

Control token ablations (NYbw W3, Yzgv W2, yrkg W1): We added new control experiments to show different insertion policies for our control tokens. We compare our learned adaptive sensor against random, periodic, fixed-boundary, and unconditional insertion in Appendix A.7 (Table 9), along with SFT-only versus SFT+GRPO training comparisons in Appendix A.7 (Table 10). We find that our learned sensor significantly outperforms these alternative control token insertion policies, and that the GRPO training step is necessary for improved reasoning performance.

Differentiation from soft prompting and activation steering (Yzgv W3, yrkg W2): We have updated the Related Work section to contrast our reflexive reasoning approach against prefix-only soft prompts and more recent control-based activation steering approaches such as CAST (Lee et al., ICLR 2025) and feedback-controller steering (Nguyen et al., ICLR 2026). We would particularly like to highlight here that our approach is the only continuous-token closed-loop control system that we are aware of, and this adaptive token-based steering system leads to substantial gains on real-world reasoning benchmarks such as GSM8K for math and CRUXEval for coding. We strongly believe that leveraging the inherent temporal nature of the LLM, with properly timed injections of learned tokens as the steering mechanism instead of blanket activation patching, allows us to achieve such significant performance gains on challenging reasoning benchmarks, beyond simple refusal and tone evaluations as in prior works.

Individual reviewer points

Yzgv (Q1): We added prefix-level AUC clarification alongside failure prediction from prompts only and matched-prompt rollouts in Section 4.1.

Yzgv (W3): We added cross-benchmark generalization results across GSM8K, PrOntoQA, and CRUXEval in Appendix A.8 (Table 11).

Yzgv (W4): We added sensitivity analyses across sensor thresholds and control token counts, and across multiple different seeds, in Appendix A.9 (Tables 12–13).

Yzgv (Ethics): We added an Ethics statement discussing potential misuse of our reflexive reasoning approach as a general-purpose latent steering method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.