- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether model-internal activation summaries from a frozen Large Language Model (Qwen3.5-4B) can provide useful action-failure uncertainty signals to guide scientific agents operating under strict evaluation budgets. The authors propose a framework () that sequentially generates two candidate proposals (), extracts public state/output features () alongside 128 layer-wise MLP activation statistics and 2 auxiliary measurements () at prediction positions, and fits a regularized logistic regression failure predictor offline. The agent then selects and executes the candidate with the lower predicted failure risk. The framework is evaluated across Materials Discovery Environments (MADE) and DiscoveryWorld benchmarks.
Critical strengths
The core hypothesis that leveraging model-internal activation uncertainty to pre-screen candidate actions and guide scientific agent decision-making, is highly intuitive, timely, and well-motivated. Scientific experimentation and high-fidelity simulations operate under strict evaluation budgets where executing failed or invalid actions incurs high computational or physical costs and delays feedback. Utilizing internal activation statistics (which are computed during standard forward passes) as a lightweight, pre-execution failure filter to avoid wasted evaluations represents a logical and elegant extension of representation probing to sequential decision-making. This strong conceptual framing provides a clear rationale for investigating whether local uncertainty signals can enhance agent efficiency.
Critical weaknesses
-
W1. Insufficient Problem Scoping and Benchmark Context for General Readers. The problem setting and environmental task dynamics are insufficiently scoped and explained for a general machine learning audience. The manuscript presumes substantial prior familiarity with the Materials Discovery Environments (MADE) and DiscoveryWorld benchmark suites. Key details such as what specific scientific objectives the agent is optimizing, what constitutes a discrete "action failure", how state transitions occur, and how environment feedback is provided, are not self-contained in the main text. A clearer, standalone description of the task formulation, environment mechanics, and evaluation protocols early in the paper is necessary to make the work accessible to broader readers.
-
W2. Presentation and Prose Need Polishing. While the underlying technical concept is sound, the manuscript's writing requires additional polishing. Certain sections suffer from awkward phrasing, dense or inconsistent notation, and redundant descriptions across methodological sections. Streamlining the narrative flow, clarifying mathematical notation, and polishing the overall prose throughout would significantly enhance readability and presentation quality.
Questions for authors
Please see Weaknesses.