acceptodds
Under review as a conference paper at ICLR 2027

How Stable Are Answers to Unresolved Scientific Questions? Separating Repeat Variability from Input Sensitivity

Abstract

When a language model gives different explanations for the same unresolved scientific question, does the difference reflect a change in the input or variation across repeated generations? Without an accepted complete explanation, disagreement alone cannot distinguish these possibilities. We introduce Science80, a controlled study of 80 questions across five scientific domains, and compare evidence reordering, deletion, and addition against responses to identical prompts. GPT-6 Astra xhigh provides the main experiment, with same-question comparisons from Gemini and Claude. Under identical inputs, the question-weighted rate of substantive revision is already 58.3%, including changes in secondary mechanisms, scope, or scientific preference. Evidence reordering does not show a significant average increment beyond repetition, whereas adding a statement increases the substantive-revision rate by approximately 20 percentage points. This increment remains after adjustment for response length and other text features. Further analyses show that a change in the preferred explanation can accompany little change in reported confidence, while limited directional evidence constrains conclusions about position and conflict. Our results support evaluating input responsiveness relative to repeated-generation variability, while assessing explanatory stability, evidence responsiveness, and scientific correctness separately.rrectness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, scientific reasoning, answer consistency, prompt sensitivity, evidence perturbations, model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ao1uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Science80, a controlled study of the stability of LLM-generated explanations for unresolved scientific questions. The benchmark contains 80 questions across five scientific domains and compares repeated generations under identical prompts with interventions that reorder, remove, or add evidence. Using GPT-6 Astra xhigh as the main configuration, with additional analyses on Gemini and Claude, the authors find substantial variation even under identical inputs, little detectable average effect from evidence reordering beyond this baseline, and a larger effect from adding evidence. The paper also studies changes in preferred explanations, confidence, evidence position, and textual complexity, and argues that input sensitivity should be interpreted relative to ordinary generation variability.

Critical strengths
  1. The paper asks a simple but important measurement question. If a model already changes its explanation across repeated generations of the same prompt, then a changed response after an intervention cannot automatically be attributed to that intervention. Using same-prompt variation as an explicit baseline is a sensible design choice.

  2. The analysis is statistically careful. The paper uses question-level aggregation, accounts for repeated permutations, reports uncertainty intervals, applies multiple-testing correction, and is appropriately cautious about null results.

  3. The paper is unusually transparent about the limitations of its setup. The authors document missing outputs, configuration heterogeneity, judging procedures, and several possible confounds in considerable detail. I appreciated that many exploratory findings are explicitly presented as such rather than overstated.

Critical weaknesses
  • The main evaluation relies heavily on model-based judgments from the same model family. Astra generates the primary responses and is also used for whole-answer grading, evidence annotation, and state mapping. The high agreement among judging contexts therefore establishes internal consistency, but not necessarily validity. This matters because distinctions such as grade 1 vs. grade 2, or whether two explanations belong to the same mechanism family, can require real domain expertise. I would be much more confident in the main results with expert validation on a non-trivial subset.

  • The “More” intervention does not cleanly isolate evidence responsiveness. Appending a statement changes the scientific content, prompt length, and the position of information at the same time, and there is only one More realization per question. The reported ~20-point increase therefore shows that the bundled intervention changes responses, but not necessarily that the model is appropriately updating in response to the scientific content of the added evidence.

  • The benchmark provides rather limited support for the position and conflict analyses. Only 11 of the 80 questions contain eligible directional evidence, and only one contains support for both sides of the candidate comparison. This makes the null reordering result hard to interpret: the benchmark may simply contain too little directional evidence for ordering effects to emerge systematically.

  • The headline 58.3% revision rate uses a fairly broad notion of “substantive change.” Grade 2 includes changes to secondary mechanisms, causal details, scope, or non-primary candidates. These are meaningful changes, but they are quite different from switching the preferred explanation. Since the primary-hypothesis field changes much less often, I think the paper should more clearly separate instability of the overall explanation from instability of the model’s scientific preference.

  • The cross-model evidence is less controlled than the main Astra experiment. Gemini and Claude are used for additional same-question comparisons, but they do not receive the same full whole-answer analysis. The Claude runs also involve mixed model receipts, different reasoning-effort settings, and missing outputs.

Questions for authors
  1. Could the authors provide blinded domain-expert annotations for a representative subset of the whole-answer comparisons and report agreement with the Astra-based grades? This would materially affect my confidence in the primary endpoint.

  2. Can the authors repeat the More condition multiple times per question, and ideally compare it with a position-matched neutral addition? This would help separate semantic responsiveness from ordinary generation variability and from prompt-length or position effects.

  3. How do the main conclusions change if substantive revision is defined more strictly, for example using grades 3–4 only, or if preferred-explanation state change is used as the primary endpoint?

  4. Were the 80 questions and evidence packets independently reviewed by domain experts for scientific accuracy, unresolved status, and neutrality? If not, I would like more discussion of how benchmark-construction errors or framing choices might affect the conclusions.

2 comments
AuthorsOpenReview ↗

Response on expert validation, matched controls, and interpretation of revision

Thank you for recognizing the measurement question and the statistical design. We have added expert verification and repeated matched controls, and revised the paper to distinguish explanation revision from changes in scientific preference.

  1. Blinded expert validation (Section 5 and Appendix F). The review set contains 120 historical answer comparisons and 260 new comparisons from a fixed, score-independent selection of 20 questions, four per domain. Two experts rated the comparisons independently before seeing Astra labels. The author-reported expert-expert exact agreement is 87.89%, with weighted Cohen's kappa=0.82 (95% CI [0.76, 0.88]). Expert-Astra exact agreement is 92.63%, with weighted kappa=0.88. Binary agreement is 92.89% at Grade >=2 and 93.95% at Grade >=3.

Expert rescoring also gives a positive More-Neutral difference on the broad endpoint. In the pooled expert subset, 99/130 More ratings and 84/130 Neutral ratings reach Grade >=2, giving +11.54 percentage points (reported 95% CI [3.20, 20.00], Holm-adjusted p=0.030). The revised manuscript reports the aggregate expert results. Item-level ratings and analysis settings are needed to reproduce the agreement and uncertainty estimates.

  1. Repeated More and a matched Neutral control (Section 4, Figures 3-4). For each of the 80 Chinese questions, the new batch includes a fresh Base, three identical-input Repeat responses, four responses to the original More addition, four to a matched Neutral addition, and two Question-only responses. More and Neutral are appended in the same position, with approximate character-length matching and screening of source coverage and candidate neutrality. Chinese length ratios range from 0.9592 to 1.0476. Conditions are interleaved.

The Grade >=2 More-Neutral difference is +12.03 percentage points (95% CI [3.80, 20.15], n=79, Holm-adjusted p=0.0148). In the fixed source-supported/conditional stratum, the estimate is +13.21 points ([4.40, 22.02], n=70, p=0.0154). Thus, the positive broad-revision contrast persists with matched additions and source-quality sensitivity. Repeated generations estimate variability under each fixed statement.

  1. Stricter revision and preferred explanations (Sections 3-4/7). We now present the three outcomes separately. Under identical inputs in the original Astra batch, the rates are 58.33% for Grade >=2, 21.88% for Grade >=3, and 28.75% for preferred-state change.

In the new matched-control experiment, the More-Neutral results are: Outcome Difference in percentage points 95% CI Holm-adjusted p Grade >=2 +12.03 [3.80, 20.15] 0.0148 Grade >=3 +6.33 [-0.63, 13.29] 0.3612 Preferred-state change +3.44 [-3.75, 10.62] 0.7275

The broad endpoint provides the strongest evidence for additional explanation revision. The stricter and preferred-state intervals include zero. P06 has unresolved Neutral grades, leaving 79 paired questions for grade contrasts and 80 for state contrasts. The abstract and conclusion now distinguish these findings. Grade 2 can reflect changes to secondary mechanisms, causal detail, scope, or non-primary candidate priority. Each of these has a different meaning from replacing a preferred explanation.

  1. Question-bank verification (Section 2 and Appendix F). Science80 was assembled from scientific literature with model assistance in question wording and evidence organization. The supplement links all 80 questions to 110 recovered sources. Subsequent expert verification covered all 480 scientific-statement annotations and 16 bilingual packets, confirming 456/480 (95.00%) and 15/16 (93.75%), respectively.

These confirmation rates accompany the statement-level source audit. Construction errors could alter plausible candidates, evidence-direction labels, or measured revision. We retain original inputs and report the fixed source-quality stratum alongside the all-question analysis. The results concern this curated corpus and its recorded evidence packets.

Position/conflict coverage and cross-model analysis. The historical position analysis retains its original 11 eligible questions and single two-sided case, with interpretation restricted to those subsets. All three historical configurations now receive the same whole-answer rubric, yielding 1,200 Astra, 1,200 Gemini, and 1,108 Claude comparisons. Common-sample and Fable-only sensitivity analyses address recorded configuration differences.

The separate Claude Fable 5.1 follow-up on 80 questions reports a Grade >=2 More-Neutral difference of +11.25 points (95% CI [3.10, 19.40], Holm-adjusted p=0.018). This adds a second-model matched-control result, reported separately from the historical Claude batch.

Thank you for the detailed response. The additional experiments address part of my concerns. However, two important issues remain. First, the new expert and Claude follow-up results are reported only in aggregate, without item-level data, so they cannot be independently verified. Second, the positive effect is limited to the broad revision endpoint, while the stricter and preferred-state results remain inconclusive. Overall, I believe my initial rating already reflects my assessment of the paper, so I will keep my score.

Reviewer_svvXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The author raise a question about when a model is faced with open scientific question, will its answer change based on the given evidence or the change can be attributed more to the model's repeated-generation variance. The question is meaningful as recently, more and more LLM assistance is included in scientific research and how should people interact with those high variance output remains an open question. Despite the valid question, the experiment is quite preliminary with only 80 questions covering three proprietary models. And when it comes to the major part of the experiment, only GPT-6 Astra is applied. The presentation of the paper also needs substential improvement.

Critical strengths

I think the major strength of the paper is raising the question of when a model is faced with open scientific question, will its answer change based on the given evidence or the change can be attributed more to the model's own repeated-generation variance. This question is previously not well studied and becomes more and more meaningful under the time of increasing LLM involvement in open scientific questions.

Critical weaknesses

Firstly, the work focuses mostly on analysis and the analysis is quite preliminary with 80 questions and using only one proprietary model in the majority part of the experiment. The 80 questions also don't contain well-explained source.

Secondly, the result is not surprising and brings limited new knowledge. The intervention of "Shuffle" and "Less" is unsurprisingly having not many impact on the final result as most of the evidence annotated by the author is neutral to the final result of the question. The "More" intervention is helpful as many "More" statements have direct link to key mechanism of the question.

The presentation needs substential improvement. Too many defensive writing is contained in the paper, making it hard to go through the paper smoothly.

Questions for authors
  1. How is the construction pipeline of the evaluation questions and their supporting evidence? Are they annotated by human expert or obtained under a complex pipeline with proprietary model backbone?

  2. Why most of the evidnece are neutral to the question? This makes "Shuffle" and "Less" having limited impact.

  3. The "rubric" mentioned in line 119 needs more explanation. The current rubric with a 0-4 scale is not detailed enough in the main content of the paper, the judge prompt can be helpful to be provided.

  4. I wonder whether instability of answer is really undesirable under this setting. As the questions are unresolved with neutral evidences, flips between candidate mechanisms reflect reasonable uncertainty. What level of variance would the author consider appropriate and how could this be tested?

2 comments
AuthorsOpenReview ↗

Response on construction, controls, evaluation, and interpretation

Thank you for recognizing the importance of distinguishing evidence-related answer changes from repeated-generation variability. We have added matched controls and expert verification, clarified the question sources and rubric, and revised the presentation.

  1. Construction and sources (Section 2). The authors assembled Science80 from scientific papers and reviews, with language-model assistance in question wording and evidence organization. The supplementary index links all 80 questions to one or two entries among 110 recovered sources, including peer-reviewed papers, preprints, and research-institution materials. Section 2 gives examples from Nature, Science, Nature Physics, Nature Communications, and Atmospheric Chemistry and Physics. We separately screened the 320 background statements, 80 original additions, and 80 new neutral additions. Subsequent expert review confirmed 456/480 statement annotations and 15/16 bilingual packets. The collection remains a curated convenience sample, and the source-coverage results and expert confirmation rates are reported separately.

  2. Neutral evidence and the contribution of the new controls (Sections 4-5). Most original background statements describe the phenomenon or constraints shared by the candidate explanations. Neutrality is defined relative to a specified candidate pair. Of 320 statements, 306 were labelled neutral, including 74 of the 80 deleted statements. We therefore interpret Shuffle/Less within this largely background-evidence setting. The historical position/conflict diagnostics are restricted to their eligible subsets.

The new experiment directly addresses whether the original More effect exceeds the effect of appending matched neutral text. For every Chinese question, we generated four answers with the same original addition and four with a neutral addition matched in position and approximate character count, alongside a fresh Base and repeated original inputs. Astra's More-Neutral difference at Grade >=2 is +12.03 percentage points (95% CI [3.80, 20.15], Holm-adjusted p=0.0148). The separate Claude Fable 5.1 follow-up on 80 questions reports +11.25 points ([3.10, 19.40], p=0.018). This comparison adds evidence about content-related revision beyond the original Shuffle/Less analysis.

  1. Rubric and judge prompts (Section 2 and Appendices B/D/F). The main text now defines each level. Grade 0 denotes equivalent scientific content. Grade 1 covers peripheral elaboration or experimental implementation. Grade 2 covers secondary mechanisms, causal detail, scope, or non-primary candidate priority. Grade 3 covers a provisional decision or within-family preference, including a change between non-preference and provisional preference. Grade 4 covers a preferred mechanism family, dominant role, or main causal direction. The whole-answer grade uses the highest substantive level.

The supplement includes complete judging prompts and ten answer-level examples. Model identity, condition, and confidence are hidden during grading. Three judging contexts provide ratings, with an exact majority required and unresolved cases retained. Two experts independently scored the answer comparisons before seeing Astra labels. Reported weighted Cohen's kappa is 0.82 between experts and 0.88 between expert and Astra grades.

  1. Appropriate uncertainty and what the results establish (Sections 5/7). We agree that switching between plausible mechanisms can express reasonable uncertainty when evidence is insufficient. The appropriate amount of variation depends on the question, candidate explanations, and informativeness of the supplied evidence. Our operational target is the additional revision associated with a specified input change, measured against repetition and matched additions.

We now distinguish broad explanation revision, stricter revision, and preferred-explanation changes throughout the paper. For Astra, More-Neutral is +6.33 points at Grade >=3 and +3.44 for preferred-state change, with both intervals including zero. The strongest result is therefore about broad explanation revision. A suitable test compares selection of an expert-supported candidate under weak, strong, and neutral evidence against repeated-input behavior. Such evaluation requires evidence whose direction and strength are established independently.

Scale and presentation. The archived study now includes 1,312 additional Astra responses and contemporaneous whole-answer regrading of the historical Gemini and Claude outputs. The separate Claude matched-control follow-up extends the broad comparison to another model. We reorganized the text around the two controls and distinct outcomes. New workflow, prompt-design, and effect figures make the design and conclusions easier to follow.

Thank you for the detailed response. However, I will keep my overall score for several reasons. First, the author only reports aggregated results without per-item data on the additional experiments, harming reproducibility. Also, the main experiment still rely on 80 questions and models, limiting the scope. Substential revision is needed for this paper. I raise my soundness to 2 as the "matched neutral additions" in section 4 do partly solve my concerns.

Reviewer_1MLoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper asks a simple but important question: when an LLM changes its explanation for an open scientific question after we tweak the prompt, is it actually reacting to the new input, or is that just normal sampling variation across repeated runs? To test this, the authors introduce Science80, a Chinese-language benchmark of 80 unresolved scientific questions across five domains. For each question, they run GPT-6 Astra (xhigh) in 16 independent contexts: 4 identical repeats, 10 evidence reorderings, 1 evidence deletion, and 1 evidence addition. Whole answers are graded against a base run on a 0–4 scale of scientific-content change (with Grade 2 counting as a substantive revision), alongside a primary-hypothesis state coding that also compares Gemini and Claude. The main takeaways are: (1) simply repeating the exact same prompt already leads to substantive revisions 58.3% of the time (mean grade 1.773); (2) reordering the evidence or deleting a statement does not produce any significant change beyond this repeat baseline; (3) appending a single evidence statement jumps the substantive-revision rate by 19.83 percentage points—an effect that holds up even after controlling for output length, hedging, and candidate mentions; and (4) reported confidence often stays flat even when the model flips its preferred hypothesis, state-based numbers shift depending on which base run is used as the reference, and positional or conflict effects cannot really be tested because 306 of the 320 base evidence statements are neutral.

Critical strengths
  • The paper makes a timely important point to be remembered by the scientific community while reporting experimental results with LLM agents: For open scientific questions without ground-truth answers, attribution of explanatory changes to input perturbations requires comparison against the model's own repeated-generation baseline. Authors propose the paired-with-Repeat estimator (Equation 1) to incorporate this. Questions (rather than output pairs) were used as the resampling unit using 10,000 bootstrap draws, centered studentized bootstrap tests with Holm multiplicity correction across interventions, and sign-flip sensitivity analysis.

  • The five-level whole-answer rubric (grades 0–4) plus field-specific change flags (Primary, Alternative, Causal chain, Scope, Experiment) captures granularity that binary equivalence metrics miss. The ten translated case studies in Appendix D (e.g., P05, C11, C15, E12) make the grade boundaries transparent. Operational anomalies (e.g., 38 missing Claude slots, mixed Fable/Opus receipts, Astra trajectory request-target header mismatches, JSON wrapper recovery rules) are candidly disclosed in Appendix A, and full prompts, responses, labels, and offline recomputation scripts are provided.

Critical weaknesses
  • Often LLMs do caching and each query output is not truly independent; this can be detrimental to the hypothesis being tested in this paper. I could not find how caching is tackled in the experiments for this paper, including other sampling parameters like temperature, top-p, seeds.

  • The paper does not explicitly account for the interaction with the LLM's parametric knowledge. The prompt explicitly instructs the model to combine the supplied evidence with its "existing scientific knowledge" (Appendix A). Because these are well-known open scientific questions, the model is likely to possess strong priors about the candidate mechanisms. Also, the audit presented in the paper states that 306/320 base evidence statements are neutral background facts. Deleting a basic background fact does not remove it from the model's internal knowledge, which trivially explains why category:Less has a tiny effect. Similarly, adding a specific fact in category:More may simply prime the model to surface a secondary mechanism it already knew. A “question-only” baseline or an annotation of whether the deleted/added facts are novel and directional relative to the model's prior, can help separate genuine evidence updating from LMM fact recall and prompt robustness.

  • It is unclear if these findings can generalize across languages: All prompts, strict character-length caps (80/160 Chinese characters), and LLM judge evaluations are exclusively in Chinese, while primary scientific literature in pretraining corpora is predominantly in English, how do you expect the trends to translate to English?

  • The 5-level rubric relies on LLM-as-a-judge (GPT-6 Astra evaluating itself) without independent human-expert validation, this can lead to self-preference and parametric biases.

Questions for authors
  • How do you make sure that LLM caching is avoided in all parts of these experiments? Also what sampling parameters (temperature, top-p, seeds) were used across the generation and judging calls?

  • Did you evaluate a `question-only’ (zero-evidence) baseline to see if the model's distribution of hypotheses without perturbations E1 to E4 differs from Base/Repeat?

  • Were the 80 deleted statements in Less and the 80 added statements in More annotated for directionality, and how many of the Less deletions were among the 306 neutral statements?

  • Can you provide replication on a small subset of questions (e.g., 1 domain / 16 questions) translated to other languages (e.g. English) to verify that these trends hold beyond a specific language?

  • Could you provide human evaluation on a small subset of the data to validate the accuracy and reliability of the 5-level LLM-as-a-judge eval?

2 comments
AuthorsOpenReview ↗

Response on reproducibility, evidence controls, language transfer, and expert validation

Thank you for recognizing the importance of the repeat baseline. We added Question-only and matched-addition controls, an English replication, and expert evaluation.

  1. Caching and sampling parameters

We audited request records, CLI commands, available configurations and raw output events. Each planned response uses a separate context and slot identifier. Local continuation reuses a valid answer only for that same slot, rather than counting it as another generation.

Positive input-cache counters occur in 1,257/1,280 historical Astra records. These document input-prefix reuse; backend response replay and statistical independence remain unverified. Separate contexts address conversation carryover.

For generation and judging, the available records provide no recoverable numerical temperature, top-p or generation-seed settings. The supplementary runner explicitly selects Astra xhigh, ignores user configuration and leaves these sampling fields unspecified, with backend values unexposed. We report this provenance gap explicitly. Selection, scheduling and bootstrap seeds are recorded separately from model-generation seeds. Thus, our results characterize repeated calls under the recorded service configuration, with backend sampling as a reproducibility limitation.

  1. Question-only baseline and parametric knowledge

Yes. We added two Question-only responses for every question, yielding 160 responses with the same instructions and output fields but without E1–E4. Relative to contemporaneous Repeat, the increase in Grade ≥2 revisions is +32.71 percentage points (95% CI [24.16, 41.46], Holm-adjusted p=0.0004). Preferred-hypothesis changes increase by +15.83 points (CI [6.88, 25.01], adjusted p=0.0044).

We also repeated the original More statement four times per question and compared it with four repetitions of one source-screened Neutral statement matched for position, style and approximate length. More–Neutral is +12.03 points at Grade ≥2 (CI [3.80, 20.15], adjusted p=0.0148; 79 questions with usable grades). At Grade ≥3, the estimate is +6.33 points (CI [-0.63, 13.29]).

These controls measure conditional-input sensitivity and broad revision beyond matched additions. A fact’s novelty to the pretrained model remains unidentified, so explanation revision and scientifically correct belief updating remain separate claims.

  1. Directionality of actual Less and More items

We linked the annotations to the actual deletion indices. Among 80 deleted statements, 74 are neutral, three support candidate A and three support B. Among 80 original More statements, 55 are neutral, six support A, 16 support B and three have unclear direction.

Neutrality is relative to the candidate pair, and neutral statements can still affect reasoning. We call More the original-addition condition and report its composition. The 74 neutral deletions limit interpretation of the small Less effect. The original order/conflict analysis also has only 11 directional questions and one with support on both sides.

  1. English replication

We translated all 16 physics questions and collected four original-input, four More and four Neutral responses per question, totaling 192 English responses. More–Neutral is +9.38 points at Grade ≥2 (95% CI [-15.62, 32.81], adjusted p=1.0000). At Grade ≥3 it is -3.12 points (CI [-23.44, 18.75]).

The broad point estimate has the same sign as the Chinese estimate, but the wide interval leaves cross-language generalization unresolved. This is an exploratory one-domain replication. We disclose the language-specific output caps: 80/160 Chinese characters versus 60/120 English words for short/long fields. Judge instructions remain Chinese, which further limits language attribution.

  1. Independent expert validation

Two experts independently scored a fixed, score-independent sample from 20 questions, four per domain, before seeing Astra labels. The review covered 120 historical and 260 follow-up answer comparisons. The blind packet hid model, condition, confidence and previous grades and randomized answer orientation.

The aggregate expert results report 87.89% exact expert–expert agreement and weighted Cohen’s kappa=0.82 (95% CI [0.76, 0.88]). Expert–Astra exact agreement is 92.63%, with weighted kappa=0.88. Binary agreement is 92.89% at Grade ≥2 and 93.95% at Grade ≥3. These results provide independent support for the rubric on the reviewed sample. The current manuscript reports aggregate expert statistics; item-level ratings and the weighting specification are still needed to reconstruct confusion matrices and independently reproduce these estimates.

Sections 4–5 and Appendix F present these additions. Our conclusion focuses on broad explanation revision under the recorded configurations.

I thank the authors for additional experiments and detailed reply. After considering the responses and revised manuscript, I have adjusted the presentation score. I maintain my overall rating. I have no further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and additional validation

Dear Area Chair and Reviewers,

Thank you for the constructive feedback. We have provided point-by-point replies to all three reviewers. This comment summarizes the main additions and the scope of the revised conclusions.

  1. Repeated and matched controls. For each Chinese question, the follow-up includes a fresh Base, three Repeat responses, four More responses, four responses to a position- and approximately length-matched Neutral addition, and two Question-only responses. More–Neutral increases Grade ≥2 revisions by 12.03 percentage points (95% CI [3.80, 20.15], Holm-adjusted p=0.0148). The Grade ≥3 and preferred-state intervals include zero, locating the clearest effect in broad explanation revision.
  2. Expert validation. Two experts independently evaluated 120 historical and 260 follow-up comparisons from 20 questions before seeing Astra labels. Reported weighted Cohen’s kappa is 0.82 between experts and 0.88 between experts and Astra. The manuscript reports aggregate results; item-level ratings and the weighting specification remain necessary for independent reproduction.
  3. Sources and evaluation clarity. The supplementary index links all 80 questions to 110 recovered sources. We added statement-level source checks and clarified the author/model-assisted construction process. The revised presentation separates the five-level rubric, stricter revision, and preferred-state changes, with complete judging prompts and ten detailed cases in the supplement.
  4. Additional checks. Question-only responses assess sensitivity to withholding the evidence packet. The 192-response English physics study remains exploratory, with wide uncertainty intervals. A separate 80-question Claude Fable 5.1 follow-up reports a Grade ≥2 More–Neutral difference of +11.25 points (95% CI [3.10, 19.40], adjusted p=0.018).
  5. Runtime provenance. We report the available context and cache records and identify the unexposed temperature, top-p and generation-seed values. Backend sampling independence remains a reproducibility limitation. The revised paper focuses on measuring explanation changes relative to repeated generation and matched additions. It distinguishes these changes from preferred-mechanism replacement and scientifically correct updating. Detailed evidence and qualifications appear in the reviewer-specific replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.