- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Science80, a controlled study of the stability of LLM-generated explanations for unresolved scientific questions. The benchmark contains 80 questions across five scientific domains and compares repeated generations under identical prompts with interventions that reorder, remove, or add evidence. Using GPT-6 Astra xhigh as the main configuration, with additional analyses on Gemini and Claude, the authors find substantial variation even under identical inputs, little detectable average effect from evidence reordering beyond this baseline, and a larger effect from adding evidence. The paper also studies changes in preferred explanations, confidence, evidence position, and textual complexity, and argues that input sensitivity should be interpreted relative to ordinary generation variability.
Critical strengths
-
The paper asks a simple but important measurement question. If a model already changes its explanation across repeated generations of the same prompt, then a changed response after an intervention cannot automatically be attributed to that intervention. Using same-prompt variation as an explicit baseline is a sensible design choice.
-
The analysis is statistically careful. The paper uses question-level aggregation, accounts for repeated permutations, reports uncertainty intervals, applies multiple-testing correction, and is appropriately cautious about null results.
-
The paper is unusually transparent about the limitations of its setup. The authors document missing outputs, configuration heterogeneity, judging procedures, and several possible confounds in considerable detail. I appreciated that many exploratory findings are explicitly presented as such rather than overstated.
Critical weaknesses
-
The main evaluation relies heavily on model-based judgments from the same model family. Astra generates the primary responses and is also used for whole-answer grading, evidence annotation, and state mapping. The high agreement among judging contexts therefore establishes internal consistency, but not necessarily validity. This matters because distinctions such as grade 1 vs. grade 2, or whether two explanations belong to the same mechanism family, can require real domain expertise. I would be much more confident in the main results with expert validation on a non-trivial subset.
-
The “More” intervention does not cleanly isolate evidence responsiveness. Appending a statement changes the scientific content, prompt length, and the position of information at the same time, and there is only one More realization per question. The reported ~20-point increase therefore shows that the bundled intervention changes responses, but not necessarily that the model is appropriately updating in response to the scientific content of the added evidence.
-
The benchmark provides rather limited support for the position and conflict analyses. Only 11 of the 80 questions contain eligible directional evidence, and only one contains support for both sides of the candidate comparison. This makes the null reordering result hard to interpret: the benchmark may simply contain too little directional evidence for ordering effects to emerge systematically.
-
The headline 58.3% revision rate uses a fairly broad notion of “substantive change.” Grade 2 includes changes to secondary mechanisms, causal details, scope, or non-primary candidates. These are meaningful changes, but they are quite different from switching the preferred explanation. Since the primary-hypothesis field changes much less often, I think the paper should more clearly separate instability of the overall explanation from instability of the model’s scientific preference.
-
The cross-model evidence is less controlled than the main Astra experiment. Gemini and Claude are used for additional same-question comparisons, but they do not receive the same full whole-answer analysis. The Claude runs also involve mixed model receipts, different reasoning-effort settings, and missing outputs.
Questions for authors
-
Could the authors provide blinded domain-expert annotations for a representative subset of the whole-answer comparisons and report agreement with the Astra-based grades? This would materially affect my confidence in the primary endpoint.
-
Can the authors repeat the More condition multiple times per question, and ideally compare it with a position-matched neutral addition? This would help separate semantic responsiveness from ordinary generation variability and from prompt-length or position effects.
-
How do the main conclusions change if substantive revision is defined more strictly, for example using grades 3–4 only, or if preferred-explanation state change is used as the primary endpoint?
-
Were the 80 questions and evidence packets independently reviewed by domain experts for scientific accuracy, unresolved status, and neutrality? If not, I would like more discussion of how benchmark-construction errors or framing choices might affect the conclusions.
2 comments
Thank you for the detailed response. The additional experiments address part of my concerns. However, two important issues remain. First, the new expert and Claude follow-up results are reported only in aggregate, without item-level data, so they cannot be independently verified. Second, the positive effect is limited to the broad revision endpoint, while the stricter and preferred-state results remain inconclusive. Overall, I believe my initial rating already reflects my assessment of the paper, so I will keep my score.
Response on expert validation, matched controls, and interpretation of revision
Thank you for recognizing the measurement question and the statistical design. We have added expert verification and repeated matched controls, and revised the paper to distinguish explanation revision from changes in scientific preference.
Expert rescoring also gives a positive More-Neutral difference on the broad endpoint. In the pooled expert subset, 99/130 More ratings and 84/130 Neutral ratings reach Grade >=2, giving +11.54 percentage points (reported 95% CI [3.20, 20.00], Holm-adjusted p=0.030). The revised manuscript reports the aggregate expert results. Item-level ratings and analysis settings are needed to reproduce the agreement and uncertainty estimates.
The Grade >=2 More-Neutral difference is +12.03 percentage points (95% CI [3.80, 20.15], n=79, Holm-adjusted p=0.0148). In the fixed source-supported/conditional stratum, the estimate is +13.21 points ([4.40, 22.02], n=70, p=0.0154). Thus, the positive broad-revision contrast persists with matched additions and source-quality sensitivity. Repeated generations estimate variability under each fixed statement.
In the new matched-control experiment, the More-Neutral results are: Outcome Difference in percentage points 95% CI Holm-adjusted p Grade >=2 +12.03 [3.80, 20.15] 0.0148 Grade >=3 +6.33 [-0.63, 13.29] 0.3612 Preferred-state change +3.44 [-3.75, 10.62] 0.7275
The broad endpoint provides the strongest evidence for additional explanation revision. The stricter and preferred-state intervals include zero. P06 has unresolved Neutral grades, leaving 79 paired questions for grade contrasts and 80 for state contrasts. The abstract and conclusion now distinguish these findings. Grade 2 can reflect changes to secondary mechanisms, causal detail, scope, or non-primary candidate priority. Each of these has a different meaning from replacing a preferred explanation.
These confirmation rates accompany the statement-level source audit. Construction errors could alter plausible candidates, evidence-direction labels, or measured revision. We retain original inputs and report the fixed source-quality stratum alongside the all-question analysis. The results concern this curated corpus and its recorded evidence packets.
Position/conflict coverage and cross-model analysis. The historical position analysis retains its original 11 eligible questions and single two-sided case, with interpretation restricted to those subsets. All three historical configurations now receive the same whole-answer rubric, yielding 1,200 Astra, 1,200 Gemini, and 1,108 Claude comparisons. Common-sample and Fable-only sensitivity analyses address recorded configuration differences.
The separate Claude Fable 5.1 follow-up on 80 questions reports a Grade >=2 More-Neutral difference of +11.25 points (95% CI [3.10, 19.40], Holm-adjusted p=0.018). This adds a second-model matched-control result, reported separately from the historical Claude batch.