- Rating
- 4(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies three ways to use predictions of a physical system for task decisions: selecting among sampled candidates, revising an answer after receiving predicted state, and applying a task rule directly without an LLM. Experiments cover simulation-grounded welding questions and a diagnostic benchmark built from public power-plant measurements. On the power-plant benchmark, grounded selection improves accuracy from 63.8% to 81.3% using exactly the same eight candidates as self-consistency. Evidence-conditioned revision reaches 95.6%, while direct model decision reaches 95.0%. Additional analyses examine candidate coverage, predictor fidelity, and failures caused by a mismatch between the scoring rule and the task.
Critical strengths
The strongest empirical result is the comparison on CCPP: grounded selection reaches 81.3% accuracy, compared with 63.8% for self-consistency, using exactly the same eight sampled candidates. This makes the gain convincing because it comes from better selection without additional candidate generation. The direct model decision control and the analyses of predictor fidelity and score–task mismatch are also valuable. They help explain where the gains come from and where the approach falls short, while showing that some tasks can be solved by applying the task rule to the predicted state without LLMs.
Critical weaknesses
The direct model comparison is useful, but the experiments mainly cover structured tasks with fixed decision rules. Direct model decision performs better than the LLM methods in the welding study and achieves similar accuracy to evidence-conditioned revision on CCPP. The results therefore provide limited evidence about the value of these LLM methods on scientific tasks that require reasoning beyond applying a rule to predicted quantities.
The revision experiment also lacks a comparison with an LLM that receives the same predicted state in its initial prompt and answers in one step. Without this comparison, it is unclear whether drafting and revising works better than providing the evidence at the start.
Questions for authors
My questions are as follows: 1. What do the current results show about when an LLM adds value beyond the predictor and fixed task rule, given the strong performance of direct model decision? 2. How much of the revision gain comes from the revision step itself rather than access to predicted state, and what evidence supports this distinction without a single-pass comparison given the same state?
3 comments
I appreciate authors taking the time to address my concerns. I will raise my scores to 4 as I promised. However, given my limited familiarity with this field, I lower my confidence score.
Thank You to Reviewer ca4Z
Thank you for reading our response and for raising your rating. We are glad that your concerns have been addressed. We remain available for any further question before the discussion closes.
Response to Reviewer ca4Z
We thank the reviewer for their comments.
W1: structured tasks and fixed decision rules
The welding and CCPP studies are the two studies in that count, and the paper now states the scope at the claim rather than only in Section 7. Revision. Section 3.4 states that every family has a specified rule, and Section 7 opens with the regime.
W2: single-pass with the same state
We ran that control. On DeepSeek-V4.1-Flash a single call with the same state reaches 153/160 (95.6%), one question above the direct rule (paired 95% CI [0.000, 0.019]); Qwen3.8-Flash-Next reaches 152/159 and GLM-5.3-Flash 152/159 (Section 5.6; Table 15). On the same 160 questions the two-call revision also reaches 153/160, so one exposure to the state recovers the same aggregate on a different model. This is a cross-model interface control, not a Kimi replication (Section 5.6; Section 1), and in this control the two-call interface adds no measurable accuracy over one exposure to the same state. Revision. Section 5.6 and Table 15 add the single-pass rows and the three model identities.
Q1: when the LLM adds value
The exception is the matched-change cell: revision leads the rule there, and that is the only one of the ten family-cells where it does. Revision. Section 5.5 reports that cell beside the rule rather than pooling it.
Q2: revision step versus access to state
This is the question answered under W2 above; the single-pass control supplies the comparison you ask for, and its result separates access to state from the second call, subject to the cross-model caveat stated there. Revision. Section 5.6 and Table 15 report that control.
You wrote that you would reconsider your rating if these concerns were addressed. The scope is now stated at the claim (Q1) and the single-pass control is in Table 15 (W2 and Q2); we ask whether that meets the condition you set.