acceptodds
Under review as a conference paper at ICLR 2027

Guiding LLM Agents with Model-Internal Uncertainty for Scientific Decision-Making

Abstract

Scientific experiments can be expensive and slow to return feedback, making uncertainty important for deciding which actions to take under a limited evaluation budget. For large language model agents, however, it remains unclear whether internal representations provide useful uncertainty signals when labeled scientific interactions are scarce. We investigate this question using a frozen Qwen3.5-4B agent in Materials Discovery Environments (MADE) and DiscoveryWorld. Our approach fits a lightweight logistic predictor offline to estimate action-failure risk from public-state and output features augmented with internal activation summaries. These summaries are extracted at the prediction positions of generated action tokens and used to score candidate actions before execution. Four experimental conditions separate single-proposal planning, two-proposal generation, public-feature risk selection, and selection augmented with internal features. Exploratory development comparisons that preserve the public-feature transformation show improved failure ranking: AUROC increases from 0.5467 to 0.5622 in MADE and from 0.9667 to 0.9867 in DiscoveryWorld Proteomics. Online evaluations identify task-specific improvements in DiscoveryWorld, while analyses of matched actions and shared candidate pools distinguish predictive information from changes in action selection. The findings support internal activations as an additional source of failure-risk information in the tested settings, while showing that improved prediction alone does not ensure better scientific outcomes.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
LLM agents, Model-internal uncertainty, AI for Science, Scientific Discovery

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZfxnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether model-internal activation summaries from a frozen Large Language Model (Qwen3.5-4B) can provide useful action-failure uncertainty signals to guide scientific agents operating under strict evaluation budgets. The authors propose a framework (P+CP+C) that sequentially generates two candidate proposals (K=2K=2), extracts public state/output features (PP) alongside 128 layer-wise MLP activation statistics and 2 auxiliary measurements (CC) at prediction positions, and fits a regularized logistic regression failure predictor offline. The agent then selects and executes the candidate with the lower predicted failure risk. The framework is evaluated across Materials Discovery Environments (MADE) and DiscoveryWorld benchmarks.

Critical strengths

The core hypothesis that leveraging model-internal activation uncertainty to pre-screen candidate actions and guide scientific agent decision-making, is highly intuitive, timely, and well-motivated. Scientific experimentation and high-fidelity simulations operate under strict evaluation budgets where executing failed or invalid actions incurs high computational or physical costs and delays feedback. Utilizing internal activation statistics (which are computed during standard forward passes) as a lightweight, pre-execution failure filter to avoid wasted evaluations represents a logical and elegant extension of representation probing to sequential decision-making. This strong conceptual framing provides a clear rationale for investigating whether local uncertainty signals can enhance agent efficiency.

Critical weaknesses
  • W1. Insufficient Problem Scoping and Benchmark Context for General Readers. The problem setting and environmental task dynamics are insufficiently scoped and explained for a general machine learning audience. The manuscript presumes substantial prior familiarity with the Materials Discovery Environments (MADE) and DiscoveryWorld benchmark suites. Key details such as what specific scientific objectives the agent is optimizing, what constitutes a discrete "action failure", how state transitions occur, and how environment feedback is provided, are not self-contained in the main text. A clearer, standalone description of the task formulation, environment mechanics, and evaluation protocols early in the paper is necessary to make the work accessible to broader readers.

  • W2. Presentation and Prose Need Polishing. While the underlying technical concept is sound, the manuscript's writing requires additional polishing. Certain sections suffer from awkward phrasing, dense or inconsistent notation, and redundant descriptions across methodological sections. Streamlining the narrative flow, clarifying mathematical notation, and polishing the overall prose throughout would significantly enhance readability and presentation quality.

Questions for authors

Please see Weaknesses.

Reviewer_vn3POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Using Qwen3.5-4B as the policy model, the authors study whether the internal activations of a frozen LLM agent provide a useful signal of action-failure risk in scientific discovery tasks with limited evaluation budgets, and whether this signal can improve decision-making. At each step, the agent generates two candidate actions. For each candidate, summary statistics of MLP activations are extracted at the action-value tokens, combined with public-state features, and passed to a lightweight logistic predictor; the candidate with lower predicted risk is executed. The experiments include several controls that separate three effects: generating an additional candidate, risk-based selection using public information, and adding internal features. Failure prediction, action selection, and final scientific outcomes are evaluated separately on MADE and DiscoveryWorld. The results show that internal features add some predictive value for failure prediction, but this improvement does not reliably lead to better scientific outcomes. The authors conclude that there is a clear gap between predictive ability and decision-making benefit.

Critical strengths

1.The experimental controls are well designed. The four arms (Single, Two, P, and P+C) separate the effects of generating an additional candidate, risk-based selection using public information, and adding internal features. This distinction is often overlooked in existing agent work. 2.The authors do not equate better failure prediction with better decisions. Instead, they evaluate prediction, action selection, and final scientific outcomes separately, and openly acknowledge that the three do not necessarily align. 3.The reporting is highly transparent. Unfavorable results, missing data, and technical reruns are all presented, and statistical intervals and computational costs are reported separately. 4.The choice of readout position is well motivated. Internal states are read only at the causal prediction positions of action-value tokens, which avoids interference from rationale and formatting tokens.

Critical weaknesses

1.Statistical reliability of RQ1. The conclusions for RQ1 currently rest on very small development sets with highly correlated samples, and the paper reports no interval estimates. Could the authors provide paired intervals for the AUROC difference between P+C and P, using cluster resampling by episode or chemical system, and report the number of positive examples in each development set? I would be more confident in this conclusion if the gain remains stable under a reasonable treatment of uncertainty. 2.Feature design and baselines. I am not convinced that 128 global mean and RMS statistics are enough to capture "internal uncertainty." Please clarify whether the MLP inputs are read before or after normalization, since this directly affects how much information these statistics can carry. A linear probe on the hidden states at the same token positions, with dimensionality reduction if needed, is a natural baseline, and I would ask the authors to include this comparison. I also question the motivation for using the reference/capture logit difference as a feature, and would like to see separate ablations of the two auxiliary measurements. 3.Potential confounders. The number and content of the pooled positions vary with the action type and argument length, so these statistics may simply encode the action itself rather than the model's uncertainty. Have the authors run controls for this? For example, they could test whether residual information remains after regressing out action type and length, or extract the same statistics from a randomly initialized model. Without such controls, it is difficult to attribute the gain to "internal uncertainty." 4.Independence of the evaluation. The paper notes that some evaluation instances were examined during development. Please clarify whether the RQ1 development data come from the same chemical systems or worlds used in the online evaluation. Please also state whether design decisions, such as the position rules and the regularization grid, were made after these instances had been examined. This determines whether the online results can be treated as a held-out test. 5.Validity of action selection. Selection quality ultimately depends on whether the two candidates in the same state are ranked correctly, but the paper never measures this directly. Since the MADE oracle is a computational potential, it should be technically feasible to compute labels offline for unexecuted candidates, and I encourage the authors to add this analysis. How do the authors explain the finding that the score-shuffle control outperforms P+C? Would the comparison between P+C and Two change with a larger number of candidates K or more diverse sampling?

Questions for authors

1.Statistical reliability of RQ1. The conclusions for RQ1 currently rest on very small development sets with highly correlated samples, and the paper reports no interval estimates. Could the authors provide paired intervals for the AUROC difference between P+C and P, using cluster resampling by episode or chemical system, and report the number of positive examples in each development set? I would be more confident in this conclusion if the gain remains stable under a reasonable treatment of uncertainty. 2.Feature design and baselines. I am not convinced that 128 global mean and RMS statistics are enough to capture "internal uncertainty." Please clarify whether the MLP inputs are read before or after normalization, since this directly affects how much information these statistics can carry. A linear probe on the hidden states at the same token positions, with dimensionality reduction if needed, is a natural baseline, and I would ask the authors to include this comparison. I also question the motivation for using the reference/capture logit difference as a feature, and would like to see separate ablations of the two auxiliary measurements. 3.Potential confounders. The number and content of the pooled positions vary with the action type and argument length, so these statistics may simply encode the action itself rather than the model's uncertainty. Have the authors run controls for this? For example, they could test whether residual information remains after regressing out action type and length, or extract the same statistics from a randomly initialized model. Without such controls, it is difficult to attribute the gain to "internal uncertainty."

Reviewer_6fo2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether internal activation summaries help a frozen Qwen3.5-4B agent predict action failure and select between two candidate actions in scientific environments. A logistic predictor uses either public-state and generated-output features (P) or an augmented representation (P+C) containing pooled MLP statistics and two auxiliary measurements. Single-proposal and risk-independent two-proposal arms control additional generation. MADE and DiscoveryWorld provide the principal online experiments, and Appendix G adds ScienceWorld. The study finds some predictive improvements and changes in action selection, but does not establish a reliable aggregate scientific-outcome advantage. The most interpretable MLP-specific comparisons are development-only; the original held-out and online evaluations test a broader feature-and-predictor bundle. Its contribution is therefore a bounded empirical investigation of this distinction, not a validated general improvement in scientific decision-making.

Critical strengths
  1. The four-arm design addresses meaningful confounds. Controlling proposal count and comparing against public/output features is more informative than evaluating an internal predictor alone. The paper additionally supplies nested development controls and genuine held-out ScienceWorld predictions, while stating the limits of each comparison (Sections 3–4; Tables 1 and 8; Appendix G).

  2. The paper preserves negative outcomes and inconvenient implementation details. It reports ranking improvements alongside deteriorating probability-quality metrics, distinguishes original from recovered runs, and does not assign outcomes to unexecuted actions. This makes the limited empirical findings more useful and inspectable than a selective performance narrative (Tables 1–3; Appendices D–G).

Critical weaknesses
  1. The methodological novelty is limited, and the comparison with direct agent studies is incomplete. Semantic Entropy Probes [1], already cited, estimates semantic uncertainty from one generation’s hidden states and examines layer and token-position choices. Doomed from the Start [2] compares internal probes, specified observable features and their combination, then evaluates risk-based episode stopping through offline replay. A lightweight predictor beside a fixed model, or an advantage over the tested observable features, is therefore insufficient as a standalone novelty claim.

From Actions to Understanding [3] already probes success/failure representations in ScienceWorld and ALFWorld and explores preliminary steering. Its task-fine-tuned policy and future-rollout labels differ from the present frozen policy, executed-action feedback and candidate selection. When Agents Commit Too Soon [4] also separates its internal consistency signal from correctness and control utility; its mixed results do not establish that internal signals are universally useless. These studies narrow the methodological novelty without fully subsuming the present setting.

The defensible increment is a controlled empirical study under specified offline supervision budgets, with a common proposal procedure and separate analysis of prediction, action changes and scientific return. Limited-label efficiency has not been demonstrated. The paper should identify what its actual findings add to [2]–[4], beyond transferring familiar probes to another setting. Those three works are absent from the supplied manuscript; [1] is not an omitted citation. This is a fixable positioning issue, not evidence of misconduct.

The remaining representation claim is also preliminary. Nested MLP controls use only 100 MADE development records from two chemistries and 40 DiscoveryWorld records from two episodes. AUROC rises by 0.0155 and 0.0200, while DiscoveryWorld NLL worsens from 0.2034 to 0.2940. These heads were not deployed online. The genuine ScienceWorld held-out evidence evaluates the original bundle, which also changes auxiliary inputs, prototype summaries and fitted parameters. A matched standard hidden-state probe is missing. Thus, the evidence for the bundle is stronger than the evidence for an independently useful MLP representation (Section 4.2; Tables 1 and 8; Appendix G).

  1. The paper gives an incomplete answer to its own decision-making question. Aggregate P+C–P scientific-outcome intervals include zero; amended standard-DiscoveryWorld differences are +3.5417 points on Easy and −1.5986 on Normal. The general prediction-to-utility gap has precedent in [4]. These scientific-task findings can still be valuable, but the accompanying analysis does not establish when internal risk estimates are useful or distinguish poor candidates, incorrect within-state ranking, and a failure target that is misaligned with scientific progress. The 456 changed selections among 1,559 eligible pools show that the selector can alter behavior, without showing that it chooses better actions. DiscoveryWorld targets immediate execution failure, while ScienceWorld uses a narrower response-message proxy. The resulting empirical contribution remains limited: the paper documents a gap between prediction and decision quality more convincingly than it explains that gap. Stronger diagnostic evidence, rather than additional formalism or repeated qualifications, is needed to make this result broadly informative (Sections 3.1 and 4.3–4.4; Table 3; Appendix G.1).
Questions for authors
  1. Contribution and prior work: The manuscript already cites Semantic Entropy Probes [1]. How do its achieved findings extend the agent comparisons in [2], the scientific-environment probing and preliminary intervention in [3], and the distinction between internal signals and control utility in the June 2026 v1 of [4]? Please compare prediction targets, policy training, supervision budgets, control mechanisms and evaluated outcomes. These works do not fully subsume the present setting, but which new finding, beyond adopting a different setting, constitutes the principal empirical contribution? Please add [2]–[4] where relevant and align the introduction with that narrower contribution.

  2. Attribution under matched supervision: Could ScienceWorld-specific nested P+MLP, P+auxiliary and P+MLP+auxiliary heads be fitted on its own training/calibration split, preserving the complete P transformation, and evaluated alongside a standard hidden-state probe on the same held-out actions and labels? Please report AUROC, NLL/Brier and task-family-level uncertainty. The unresolved question is whether the attributable MLP increment survives beyond the small development sets. If limited-label efficiency is intended as a method advantage, please provide label-budget sensitivity grouped by independent tasks or episodes. Otherwise, present limited supervision as the experimental setting rather than an established comparative advantage.

  3. Within-state ranking and scientific value: Is there an independently labeled candidate-ranking set or a small restored-state subset in which both proposals can be executed? Could it distinguish inadequate candidate sets, incorrect risk ranking and disagreement between immediate failure and scientific progress? The shared-pool counts establish behavioral influence, not better selection. This is a request for additional mechanism evidence; if unavailable, the contribution should remain explicitly descriptive rather than implying that the conditions for useful scientific guidance have been identified.

  4. Failure-label coverage: Can existing logs quantify how often excluded parser errors such as 'Unknown action.' occur in ScienceWorld, and how often DiscoveryWorld actions labeled non-failure execute without scientific progress? Please report what the current target actually covers and explain which conclusions survive that restriction. Alternative-label sensitivity analysis would be additional validation, distinct from clarification of the existing labels (Section 3.1; Appendix G.1).

  5. Reproducibility and verification: The AI assistance statement covers research design, code, tests, orchestration, analysis, literature checking, figures and writing. Please specify the checks used to verify generated code, labels, feature extraction and reported statistics, and identify which supporting artifacts are available. The reproducibility statement says that not every raw trajectory, checkpoint or simulator is distributed. Which principal results can an independent reader actually reproduce from the supplied package, and which require additional material? This asks about verification and reproducibility, not about inferring authorship from prose.

  6. Bibliographic corrections: Please complete the proceedings metadata for Transcoders and DiscoveryWorld (NeurIPS 2024; DiscoveryWorld in the Datasets and Benchmarks track) and ReAct (ICLR 2023), using metadata appropriate to the cited versions. For Reflexion, clarify the intended version and reconcile its author list with that choice: the arXiv record includes Edward Berman, whereas the NeurIPS 2023 proceedings list five authors. Retaining arXiv access links is acceptable. These are publication/version consistency corrections and do not independently determine the scientific recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.