acceptodds
Under review as a conference paper at ICLR 2027

From Sampling to Selection: Grounding LLM Reasoning with Predictive Models of Physical Systems

Abstract

Test-time sampling can place a correct answer in the candidate pool without providing a reliable way to identify it. Scientific workflows offer an additional source of evidence: predictive models of the physical system. We compare three placements of the same predictive-state evidence during LLM inference—post-generation candidate selection, one-step evidence-conditioned revision, and direct state-to-decision mapping when a task rule is specified—and ask whether such evidence helps and when it makes the language model unnecessary. In a controlled simulation-grounded study and an independently implemented diagnostic study on public combined-cycle power-plant data, candidate coverage far outruns majority voting: with eight matched samples a correct candidate is present for 91.3% of questions, while self-consistency returns 63.8% and predictive-state selection 81.3%, recovering 63.6% of the selection gap at no additional sampling cost. Two controls delimit the mechanism. Selection fails when the score evaluates the wrong task quantity and, more stringently, when the required state change is smaller than the predictor's own error. And when the task's answer is a deterministic function of the predicted state, applying the task rule directly to that state (95.0%) matches the best LLM interface (95.6%) with no language-model calls at all. The results give a conditional account of predictive-state grounding and a concrete criterion for when language inference adds value over the predictor alone.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
LLM agents, scientific agents, scientific code generation, neuro-symbolic AI, scientific machine learning, verification, scientific simulation, specification grounding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ca4ZOpenReview ↗
Rating
4(+1)
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies three ways to use predictions of a physical system for task decisions: selecting among sampled candidates, revising an answer after receiving predicted state, and applying a task rule directly without an LLM. Experiments cover simulation-grounded welding questions and a diagnostic benchmark built from public power-plant measurements. On the power-plant benchmark, grounded selection improves accuracy from 63.8% to 81.3% using exactly the same eight candidates as self-consistency. Evidence-conditioned revision reaches 95.6%, while direct model decision reaches 95.0%. Additional analyses examine candidate coverage, predictor fidelity, and failures caused by a mismatch between the scoring rule and the task.

Critical strengths

The strongest empirical result is the comparison on CCPP: grounded selection reaches 81.3% accuracy, compared with 63.8% for self-consistency, using exactly the same eight sampled candidates. This makes the gain convincing because it comes from better selection without additional candidate generation. The direct model decision control and the analyses of predictor fidelity and score–task mismatch are also valuable. They help explain where the gains come from and where the approach falls short, while showing that some tasks can be solved by applying the task rule to the predicted state without LLMs.

Critical weaknesses

The direct model comparison is useful, but the experiments mainly cover structured tasks with fixed decision rules. Direct model decision performs better than the LLM methods in the welding study and achieves similar accuracy to evidence-conditioned revision on CCPP. The results therefore provide limited evidence about the value of these LLM methods on scientific tasks that require reasoning beyond applying a rule to predicted quantities.

The revision experiment also lacks a comparison with an LLM that receives the same predicted state in its initial prompt and answers in one step. Without this comparison, it is unclear whether drafting and revising works better than providing the evidence at the start.

Questions for authors

My questions are as follows: 1. What do the current results show about when an LLM adds value beyond the predictor and fixed task rule, given the strong performance of direct model decision? 2. How much of the revision gain comes from the revision step itself rather than access to predicted state, and what evidence supports this distinction without a single-pass comparison given the same state?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ca4Z

We thank the reviewer for their comments.

W1: structured tasks and fixed decision rules

"The direct model comparison is useful, but the experiments mainly cover structured tasks with fixed decision rules."

The welding and CCPP studies are the two studies in that count, and the paper now states the scope at the claim rather than only in Section 7. Revision. Section 3.4 states that every family has a specified rule, and Section 7 opens with the regime.

W2: single-pass with the same state

"The revision experiment also lacks a comparison with an LLM that receives the same predicted state in its initial prompt and answers in one step."

We ran that control. On DeepSeek-V4.1-Flash a single call with the same state reaches 153/160 (95.6%), one question above the direct rule (paired 95% CI [0.000, 0.019]); Qwen3.8-Flash-Next reaches 152/159 and GLM-5.3-Flash 152/159 (Section 5.6; Table 15). On the same 160 questions the two-call revision also reaches 153/160, so one exposure to the state recovers the same aggregate on a different model. This is a cross-model interface control, not a Kimi replication (Section 5.6; Section 1), and in this control the two-call interface adds no measurable accuracy over one exposure to the same state. Revision. Section 5.6 and Table 15 add the single-pass rows and the three model identities.

Q1: when the LLM adds value

"What do the current results show about when an LLM adds value beyond the predictor and fixed task rule, given the strong performance of direct model decision?"

The exception is the matched-change cell: revision leads the rule there, and that is the only one of the ten family-cells where it does. Revision. Section 5.5 reports that cell beside the rule rather than pooling it.

Q2: revision step versus access to state

"How much of the revision gain comes from the revision step itself rather than access to predicted state, and what evidence supports this distinction without a single-pass comparison given the same state?"

This is the question answered under W2 above; the single-pass control supplies the comparison you ask for, and its result separates access to state from the second call, subject to the cross-model caveat stated there. Revision. Section 5.6 and Table 15 report that control.

You wrote that you would reconsider your rating if these concerns were addressed. The scope is now stated at the claim (Q1) and the single-pass control is in Table 15 (W2 and Q2); we ask whether that meets the condition you set.

I appreciate authors taking the time to address my concerns. I will raise my scores to 4 as I promised. However, given my limited familiarity with this field, I lower my confidence score.

AuthorsOpenReview ↗

Thank You to Reviewer ca4Z

Thank you for reading our response and for raising your rating. We are glad that your concerns have been addressed. We remain available for any further question before the discussion closes.

Reviewer_vZMTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Given candidate LLM answers and an independently trained predictor z of physical state, the paper compares three placements of z: hidden post-hoc candidate selection ("grounded selection", a hand-designed score ranks sampled candidates), visible one-step revision, and a no-LLM direct rule applied to z. Experiments: a welding study (139 questions, CFD labels, learned surrogate, four LLMs) and a CCPP study (160 questions, random-forest predictor, Kimi K3 without thinking). On CCPP at N=8, a correct candidate is present for 91.3% of questions, self-consistency reaches 63.8%, grounded selection 81.3% (recovering 63.6% of the gap), revision 95.6%, and the direct rule 95.0%. Controls show failure when the score evaluates the wrong quantity (matched-change questions, 50%) and when the required change is below predictor error; a post hoc direction-agreement score raises matched-change to 71.9%.

Critical strengths
  1. Clean matched-candidate design. Self-consistency and grounded selection reuse identical candidate pools at each N, so the 17.5-point gain isolates the selection signal and is not due to extra LLM calls (token accounting provided). The coverage-vs-selection gap (91.3% vs. 63.8%) is an informative way to frame test-time scaling.
  2. Honest negative controls and transparency. The matched-change failure is reported and analyzed, the four-condition fidelity ordering is acknowledged as not a scaling law, the CCPP protocol is frozen with paired bootstrap CIs and integrity checks, and the limitations section is accurate.
Critical weaknesses
  1. The central message is largely implied by the task construction, and the headline method is dominated. Answers are deterministic functions of the predicted state (2% numerics, thresholds, pairwise, constrained selection), so a good predictor helps and the no-LLM rule matches the LLM (95.0% vs. 95.6%); the interesting regime (rule not a function of z) is outside the study. Revision reaches 95.6% with 2 calls (about 1.2k tokens) versus 81.3% with 8 calls (about 3.9k tokens) for grounded selection, so "no additional sampling" matters only if one has already committed to 8 samples, and no data show when hidden selection is preferable. In the welding study the surrogate/simulation come from the label-defining simulation and the LLM-only baseline cannot know CFD outputs, so the gap measures access to information.
  2. The selection baseline is weak and the key failure analysis is post hoc on the evaluation items. How self-consistency aggregates free-form floats (2% tolerance) and selection answers is not stated (SC is 50% on numerics), and no median-of-N or LLM-verifier/best-of-N baseline is run at equal budget; the score S = s_p + 1.5 s_t is hand-set. The direction-agreement score (matched-change 50% to 71.9%, overall 81.3% to 88.1%, r = 0.72) was devised after seeing the failure on the same 32 questions/pools, so it is not held-out evidence. Evidence is also narrow: a 4-input/1-target system, one non-thinking LLM, 32 questions per family.
Questions for authors
  1. How are SC votes aggregated for numeric and constrained-selection answers? Why is SC exactly 50.0% on the binary direction task (a systematic LLM bias?)
  2. How do median-of-N and a verifier/best-of-N baseline perform at equal sampling budget?
  3. Does the direction-agreement score hold on held-out questions or another physical system, and do results hold with a thinking-enabled LLM and a second LLM on CCPP?
  4. Is there a task where the answer is not a given function of z in which the LLM adds value?
3 comments
AuthorsOpenReview ↗

Response to Reviewer vZMT (1/2)

We thank the reviewer for their comments.

W1: task construction, cost, and the welding tiers

"The central message is largely implied by the task construction, and the headline method is dominated. Answers are deterministic functions of the predicted state (2% numerics, thresholds, pairwise, constrained selection), so a good predictor helps and the no-LLM rule matches the LLM (95.0% vs. 95.6%); the interesting regime (rule not a function of z) is outside the study."

The paper states it at the claim and does not claim a rule-free result. On cost, the paper prints both currencies: where the state can be shown, revision reaches 95.6% at two calls and 1,244 tokens per question against 81.3% at eight calls and 3,914 tokens for grounded selection (Table 7; App. B.8). The cost and accuracy advantages therefore point the same way in that regime. Hidden selection is what remains when the state cannot be disclosed, and it recovers 63.6% of the available selection gap with no extra calls (Section 5.4; App. B.7); we do not claim it is preferable when the state can be shown. On welding, the surrogate and simulation tiers come from the label-defining simulation, and the LLM-only baseline cannot know the CFD outputs, so that gap measures access to information; Section 4.3 labels the simulation tier as a bound rather than a deployable signal.

Revision. Table 7 and App. B.8 report calls and tokens, App. B.7 reports the hidden-selection recovery, and Section 4.3 marks the simulation tier as a bound.

W2: selection baseline, post-hoc analysis, and evidence breadth

"The selection baseline is weak and the key failure analysis is post hoc on the evaluation items."

The aggregation and missing-baseline points are answered under Q1 and Q2 below, and the post-hoc point is answered under Q3. On the hand-set weight, λ\lambda = 1.5 sits on a flat sweep across [0.5, 3]; at λ\lambda = 0 the score falls to 70.6%, with the standalone magnitude rule at 71.9% on the same pools (App. L; Table 15). The scale is stated in Section 7 and Section 5.1: four inputs, one target, 32 questions per family, one base LLM with thinking disabled; cross-model robustness is measured in the controlled study. Revision. Section 5.1 and App. B.5 state the aggregation rule, Section 5.6 and Table 15 report the sweep and the model controls, and Section 7 states the scale.

Q1: self-consistency aggregation and the 50.0%

"How are SC votes aggregated for numeric and constrained-selection answers? Why is SC exactly 50.0% on the binary direction task (a systematic LLM bias?)"

Self-consistency takes the median of the eight numerical estimates and the plurality of the eight categorical answers, with ties to the earliest sample (Section 5.1; App. B.5); constrained selection therefore uses the plurality rule. On the numerical family that median reaches .500, and on constrained selection the plurality reaches .375 (Table 6). The matched-change 50.0% is not produced by the tie rule: a tie-policy sweep moves the four categorical families by at most 3.3 pp. Matched change is the only family where self-consistency falls below both single-call baselines, chain-of-thought .594 and direct prompting .656 (Section 5.5; Table 6). Revision. Section 5.1 and App. B.5 state the rule, and Section 5.5 reports the matched-change cell against the two single-call baselines.

AuthorsOpenReview ↗

Response to Reviewer vZMT (2/2)

Q2: median-of-N and verifier best-of-N

"How do median-of-N and a verifier/best-of-N baseline perform at equal sampling budget?"

Both are conditions of the comparison: median-of-eight on numerical questions and plurality-of-eight on categorical questions are the self-consistency arm, which reaches 63.8% overall and .500 on the numerical family (Table 5; Table 6). Verifier best-of-eight under the paper's predictor is the grounded-selection arm: 81.3% from the same eight candidates, with one predictor pass and no extra language-model calls (Section 5.2; Table 5). A separately trained verifier was not run; an independent LLM judge scoring the same eight candidates without the predicted state reaches 94/160 (58.8%), below self-consistency's 102/160 (63.8%; exact McNemar p=0.23p = 0.23) and far below predictor-grounded selection's 130/160 (p=4.4×10−8p = 4.4\times10^{-8}). Revision. Section 5.1 names both aggregation rules and Table 6 reports the per-family cells.

Q3: held-out direction score, second system, and model controls

"Does the direction-agreement score hold on held-out questions or another physical system, and do results hold with a thinking-enabled LLM and a second LLM on CCPP?"

The direction score is post-hoc in origin: on the 32 questions used to design it, it raises matched-change selection from 50.0% to 71.9%, and that number is not held-out evidence (App. D). On 32 fresh questions drawn from rows not used to design the score, the same score selects correctly in 26/32 = 81.3%, against 23/32 = 71.9% for a magnitude-only rule, 22/32 = 68.8% for majority vote, and 71.5% for a uniform-random candidate (App. D; Table 15). That fresh set is the only held-out estimate for the score, and it is still the same physical system; we did not test the score on a second system. The 88.1% figure is the same score applied in-sample to all 160 questions, not a held-out result (App. D). On the model question, the pool ordering reproduces on DeepSeek-V4.1-Flash: aligned selection 80.6%, self-consistency 55.6%, direct 52.5% (Table 15). Frozen-score selection on that model is 66.3% at 82.5% coverage, so the aligned score moves the ordering rather than a different candidate pool. We did not add a thinking-enabled condition to the revision: the CCPP protocol uses Kimi K3 with thinking disabled (Section 5.1), so the ordering is not claimed under thinking. Revision. App. D reports the design-set and fresh-set numbers separately, and Table 15 labels the score post-hoc in origin.

Q4: a task where the answer is not a function of z

"Is there a task where the answer is not a given function of z in which the LLM adds value?"

We did not test such a task: every family in both studies is a deterministic function of predicted state and a specified rule, and Section 7 states this. The closest cell is matched change, where revision leads the rule, and even there the rule is specified. Revision. Section 7 states the boundary at the start rather than at the end.

AuthorsOpenReview ↗

Follow-up to Reviewer vZMT

We would like to follow up on our rebuttal before the author-reviewer discussion closes on Nov. 18 at 11:59 p.m. AoE. The rebuttal answers each of your comments in detail. In particular, it covers the held-out test of the direction score that you asked for. On 32 fresh questions the score selects correctly in 26/32 = 81.3%, against 71.9% for a magnitude-only rule and 68.8% for majority vote (App. D, Table 15). It also covers the second LLM. The pool ordering reproduces on DeepSeek-V4.1-Flash, with aligned selection at 80.6% against 55.6% for self-consistency (Table 15). The cost of each placement is now reported in calls and tokens (Table 7, App. B.8), and the aggregation rule of self-consistency is stated in Section 5.1. We hope these additions give a fuller picture of the work.

We would also like to highlight the main result of the paper. Sampling several answers from an LLM and voting, as in self-consistency, is already a standard way to scale reasoning at test time, and verifier-based selection works where a checker or labeled solutions exist. Questions about a physical process have neither. Our paper shows that a predictive model of the process can supply the selection signal. On the same eight candidates, grounded selection reaches 81.3% against 63.8% for self-consistency on the power plant (a 17.5-point gain, paired 95% CI [10.6, 24.4]), with no extra LLM call. On welding it reaches 75.5% against 36.7%. These results are, to the best of our knowledge, new in the literature on test-time scaling for physical systems. Your review notes that the matched candidate pools isolate the selection signal, and we hope the new controls complete this picture. Reviewer ca4Z has raised the rating from 3 to 4 after our response.

We would be very grateful if you could read the rebuttal before the discussion closes and tell us what you think. If it resolves your concerns, we kindly ask you to consider raising your score. If anything remains open, please let us know and we will answer right away.

Thank you for your time and consideration. We look forward to your reply.

Reviewer_HF5uOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper is an empirical investigation of how predictions from physical models can improve LLM reasoning at inference time. The authors compare three ways of using the same predicted state: ranking sampled candidates against it without showing it to the LLM (grounded selection), showing it to the LLM for one revision step, and applying the task’s decision rule directly to the predicted state without the LLM.

Critical strengths

S1. The paper uses a clean experimental design, particularly the matched candidate pools that isolate the effects of answer selection from sampling.

S2. The paper acknowledges the limitations of its approach and reports informative negative results, including cases where the verification score does not align with the task.

Critical weaknesses

W1. The study is limited to settings where the language model appears largely unnecessary, which weakens both the motivation and the practical significance of the contribution. In the evaluated tasks, an independent predictor supplies the relevant physical state, and a specified decision rule maps that state to the answer. The paper's own experiments support this concern: applying the decision rule directly to the predicted state achieves over 90% accuracy in both studies, matching or outperforming the LLM-based approaches.

The evaluation tasks also highlight this concern. Most involve numerical predictions, comparisons, or simple decisions that can be handled directly by the predictive model. As a result, the finding that LLM inference adds little value in these settings is largely expected. More broadly, the paper does not investigate settings in which the LLM provides capabilities complementary to those of the predictor. A more compelling evaluation would involve scientific tasks where the predictive model is useful but insufficient, such as when decision rules are unspecified, constraints must be interpreted from natural language, or the predictor captures only part of the relevant physical state.

W2. The novelty of the main empirical findings is limited relative to existing research on test-time sampling and verification. Prior work has already established that repeated sampling can substantially increase candidate coverage without corresponding improvements in majority-vote accuracy [Ref1], and that verifier-based selection can outperform majority voting when reliable verification signals are available [Ref2]. The matched-candidate experimental design is informative, but primarily confirms these established findings in a new application domain.

References:

[Ref1] Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q.V., Ré, C. and Mirhoseini, A., Large language monkeys: Scaling inference compute with repeated sampling. ICLR 2025

[Ref2] Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R. and Hesse, C., 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Questions for authors

Could the authors clarify whether all tasks in both studies have answers that are deterministic functions of the predicted state? Section 3.4 introduces direct model decisions only for tasks with a specified decision rule, yet Tables 2 and 6 report results for every task family. If all tasks can be solved directly using the predictor, this raises questions about the need for LLM inference.

Could the authors evaluate settings where the predictor is useful but insufficient, such as when decision rules must be inferred from natural language or the predictor provides only part of the relevant information? Demonstrating the added value of LLM reasoning in such settings would substantially strengthen the contribution.

3 comments
AuthorsOpenReview ↗

Response to Reviewer HF5u

We thank the reviewer for their comments.

W1: settings where the language model is unnecessary

"The study is limited to settings where the language model appears largely unnecessary, which weakens both the motivation and the practical significance of the contribution."

The direct rule is the reference point of the study. It requires a decision rule to be written by hand for every question type (Table 16). Given the same predicted state, a single LLM call reaches 153/160 (95.6%) on the power-plant questions, against 95.0% for the rule (Section 5.6, Table 15). The LLM interface therefore matches the hand-written rule while taking the question in natural language. Our follow-up comment below gives the practical motivation in full. Revision. Section 3.4 and Section 7 state the scope of the claim.

W2: novelty relative to test-time sampling and verification

"The novelty of the main empirical findings is limited relative to existing research on test-time sampling and verification."

Prior work shows that repeated sampling raises coverage and that a reliable verifier selects better than a majority vote. Both results need a reliable verification signal. Questions about a physical process have no checker and no labeled solutions to train one. Our contribution is the matched-pool comparison of three placements plus a direct-rule control in every family, with an independent physical predictor as the only verification signal (Section 5.2; Section 5.6). Section 5.4 reports the coverage gap (91.3% against 63.8%) on these pools, and Sections 4.3 and 5.5 measure how predictor quality and the choice of score limit the selection. Revision. Section 1 and Section 2 place the contribution against test-time sampling and verifier selection.

Q1: deterministic answers and rule provenance

"Could the authors clarify whether all tasks in both studies have answers that are deterministic functions of the predicted state? Section 3.4 introduces direct model decisions only for tasks with a specified decision rule, yet Tables 2 and 6 report results for every task family."

Yes. Every family in both studies has a specified decision rule, so each answer is a deterministic function of predicted state and the question; Section 3.4 now says that unconditionally, and Table 16 lists the ten rules. Five CCPP rules are executable in the released code; the five welding rules are reconstructed from the task descriptions and are not code-verified, as App. M states. That is why the direct rule is a control in every family: it measures what the LLM interface adds when the rule and state already determine the answer. Revision. Section 3.4 states the unconditional version, and Table 16 and App. M give the provenance.

Q2: predictor useful but insufficient

"Could the authors evaluate settings where the predictor is useful but insufficient, such as when decision rules must be inferred from natural language or the predictor provides only part of the relevant information?"

Every tested family has a specified rule, and Section 7 states this scope. Within it, the matched-change family is the closest case to a predictor that is useful but insufficient. The required state change (median 3.65 MW) is close to the predictor error (2.41 MW), and the LLM revision reaches .875 there against .844 for the direct rule (Section 5.5; App. D). Revision. Section 7 states the scope at the start.

AuthorsOpenReview ↗

Follow-up to Reviewer HF5u: data-limited setting and novelty

We would like to add three points to our earlier response, on the practical value of this study when data are scarce, on the role of the LLM, and on novelty.

1. Accurate answers when data are scarce

In industrial processes such as laser-arc welding and power generation, data are hard to obtain. Each new data point is a physical trial or a high-fidelity CFD run. Engineers still have to answer many what-if questions about operating conditions. Does a setting cross a temperature limit? Which of two settings gives the deeper melt pool? Which configuration meets a target output? An LLM is the natural interface for such questions, but on its own it is unreliable here. Chain-of-thought and self-consistency reach 36.0% and 36.7% on the welding questions (33.8-41.0% across four LLMs) and 59.4% and 63.8% on the power-plant questions. More sampling does not close this gap.

Our approach is designed for this data-limited setting, and it gives accurate answers there.

  • It needs no task-specific training data. The LLM is used as it is. No fine-tuning, no verifier training and no labeled reasoning traces are involved. The only added component is a predictive model of the process, which such a workflow already has.
  • The answers become accurate. With evidence-conditioned revision, accuracy rises from 36.7% to 84.9% on welding and from 63.8% to 95.6% on the power plant. Selection reaches 75.5% and 81.3%, and it reuses the same eight samples, so its gain costs one predictor call.
  • The predictor can be simple. On welding, a trilinear interpolation of existing simulation points already lifts selection from 36.7% to 61.9%. A learned surrogate gives 75.5%, and the exact simulation state gives 79.9% (Section 4.3). On the power plant, a linear model with R2 = 0.928 still gives +16.9 points over self-consistency, against +24.4 for the random forest (Appendix D). Most of the benefit is available with a low-cost predictor, which is what a data-limited setting can afford.

A question answered this way needs no new trial and no new simulation. This is the saving in experimental cost and development effort that motivates the study.

2. The role of the LLM

The direct rule is the reference point of the study. It requires a decision rule to be written by hand for every question type (Table 16). The LLM interface takes the question in natural language. Given the same predicted state, a single LLM call reaches 153/160 (95.6%), against 95.0% for the hand-written rule, and two further models reach 152/159 (Section 5.6, Table 15). The LLM interface therefore matches the hand-written rule without a rule being coded for each task.

The study also tells a practitioner where the evidence should enter. Visible evidence works best for numerical questions. Hidden selection needs a score that matches the decision, and Section 5.5 shows how to detect and repair a mismatch (50.0% -> 71.9%, and 81.3% on 32 held-out questions).

3. Novelty relative to [Ref1] and [Ref2]

Both references rely on a reliable verification signal. In [Ref1] the tasks with large gains have automatic verifiers such as unit tests and proof checkers. In [Ref2] the verifier is trained on labeled solutions from the same task distribution. Questions about a physical process have neither. There is no checker, and there are no labeled solutions to train one. Our study asks whether a predictive model of the physical system can supply the signal. This model was never trained on LLM outputs, and it is itself imperfect. To our knowledge three results are new.

  • The same evidence is compared in three placements on identical candidate pools, with a no-LLM control in every task family. This separates the value of the evidence from the value of the language model (Sections 5.2 to 5.4).
  • The verifier is noisy, and we measure what this costs. The fidelity sweeps (Section 4.3, Appendix D) give the dependence on predictor quality. The noise-floor case gives the limit: selection stops helping when the required state change (3.65 MW) is close to the predictor error (2.41 MW).
  • A failure mode that is specific to this kind of verifier. A score that checks the wrong physical quantity has a correlation of -0.05 with correctness, while a task-aligned score has +0.72 (Section 5.5).

We hope this clarifies the motivation and the contribution. If it addresses your concerns, we would appreciate it if you would consider updating your score.

AuthorsOpenReview ↗

Reminder to Reviewer HF5u

As the author-reviewer discussion closes on Nov. 18 at 11:59 p.m. AoE, we would like to follow up on our response and on the follow-up above. They address the practical value of the study when data are scarce, the role of the LLM, and the novelty relative to [Ref1] and [Ref2]. Could you please let us know if you have any other questions or concerns?

Reviewer ca4Z has raised the rating from 3 to 4 after our response. If your concerns are resolved as well, we kindly ask you to consider raising your score.

Thank you for your time and consideration. We look forward to your reply.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their constructive feedback and valuable insights. The reviews converge on the same point: every tested task has a specified decision rule, so a direct rule is a strong baseline and the evidence is limited to that regime. The strongest new result is the single-pass control: with the same predicted state in one call, DeepSeek-V4.1-Flash reaches 153/160 (95.6%), one question above the direct rule (95.0%; paired 95% CI [0.000, 0.019]), and Qwen3.8-Flash-Next and GLM-5.3-Flash each reach 152/159 (Section 5.6; Table 15). The two-call revision also reaches 153/160 on the same 160 questions, so one exposure to the state recovers the same aggregate. Revision adds value in the matched-change cell, where it reaches .875 against .844 for the direct rule, with a median absolute change of 3.65 MW against the predictor's MAE of 2.41 MW (Section 5.5; App. D). The CCPP placement ordering - grounded selection at 81.3% against 63.8% for self-consistency on the same eight candidates - is a separate comparison in Section 5.2. Revisions are marked in blue in the revised PDF.

AuthorsOpenReview ↗

Summary of the Rebuttal for the Area Chair and Reviewers

We thank the Area Chair and the reviewers for their time and valuable feedback. As the discussion period is ending, we thought it could be useful to summarize the current state of the rebuttal.

We would first like to highlight the main result of the paper. Sampling several answers from an LLM and voting, as in self-consistency, is already a standard way to scale reasoning at test time, and verifier-based selection works where a checker or labeled solutions exist. Questions about a physical process have neither. Before this work there was, to the best of our knowledge, no test of whether a predictive model of the physical system can supply this signal. Our paper provides this test. On the same eight candidates, grounded selection reaches 81.3% against 63.8% for self-consistency on the power plant (a 17.5-point gain, paired 95% CI [10.6, 24.4]), with no extra LLM call. On welding it reaches 75.5% against 36.7%. All three reviews list the matched-candidate design as a strength of the paper.

We initially received scores 3, 2, 1. Based on reviewer feedback, we added the following to our manuscript:

  • A new single-pass control with the same predicted state. In one call, DeepSeek-V4.1-Flash reaches 153/160 (95.6%), within one question of the direct rule (95.0%), and Qwen3.8-Flash-Next and GLM-5.3-Flash each reach 152/159 (Section 5.6, Table 15).
  • A new held-out test of the direction score. On 32 fresh questions it selects correctly in 26/32 = 81.3%, against 71.9% for a magnitude-only rule and 68.8% for majority vote (App. D, Table 15).
  • A replication of the pool ordering on a second LLM, with aligned selection at 80.6% against 55.6% for self-consistency (Table 15).
  • A full account of the cost of each placement in calls and tokens (Table 7, App. B.8).
  • A statement of the aggregation rule of self-consistency (Section 5.1, App. B.5), and a statement of the scope of the study at the start of Section 7.

During the discussion phase, we provided detailed responses to the concerns raised by all three reviewers and revised the paper accordingly. The rating changes and current status of the three reviewers are as follows:

ReviewerOriginal RatingAfter Discussion
ca4Z34 (Nov. 18)
vZMT22 (No Reply)
HF5u11 (No Reply)

Following our additional experiments and clarifications, Reviewer ca4Z increased their score from 3 to 4. Reviewers vZMT and HF5u have not yet responded. We have addressed the reviewers' comments in our responses, including a follow-up to Reviewer HF5u regarding the data-limited setting and the novelty of our work.

In light of these updates, we hope the Area Chair will take our additional experiments and clarifications into consideration. We would also appreciate it if the reviewers could revisit our responses and update their scores if their concerns have been adequately addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.