- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces the TSC-QA collection of 200 question pairs, associated with a corpus of scholarly documents. Each sample is a question that has two contradicting answers, depending on what time cutoff we consider (e.g., something that was believed to be true at time X1, and is refuted at a later time X2). Evaluation of models is on the capability of models to answer both questions correctly in different sessions (get the early answer for an early cutoff, and the late answer with the late cutoff).
Critical strengths
- The work of time-dependence on scientific claims is interesting and timely, the dataset, built based on the citation graph, is an interesting procedure.
- The example provided in the introduction is clear and compelling, this is one of the few things that is clear in the presentation of the work.
Critical weaknesses
W1. Everything from Section 4 onwards needs to be rewritten, as it is absolutely unclear what is going on. The paper goes in very rapid succession from one experiment to the next, with barely any explanation, or interpretation of the results. For example:
- Table 1 introduces Workflow, Procedure and Group. It is unclear that this is ever used again (particularly the Group).
- Figure 3 is a sort of cherry-picked selection of Table 2, which is the main Table of results. Mind-blowingly, the draft describes the results in Table 2 for all of 1 paragraphs (lines 308-312) before moving on. I stared at the raw table for more than 5 minutes and could not understand what the main insights are that I should take away as a reader. The authors need to explain the results; this is not a dump of data, this is a paper meant for an audience who needs to understand the main critical findings.
- Lots of experimental details are actually in the Table captions (Source information / Standard vs. Guided Evidence use) Web vs. Fixed for GPT. It is not acceptable to have so little detail on the main experiment of the paper. The paper claims Table 2 contains all results, but the Codex models seem to be missing (unsure why), which points to one of many confusions.
- Tables 3,4,5 (which aren't Tables and should be figures...) are somehow rehashes (?) of the main results, averaged in an unknown way. Unclear how these are generated.
W2. The paper then goes full steam ahead and does an LLM-based analysis of reasoning traces (response analysis), with a lot of new nomenclature (Early / Late) and Low Medium High. It is unclear what the value of this analysis is, as the text mostly describes values in the Table, and does not help us interpret the results.
W3. It seems like the dataset was generated with the help of AI (LLM-based) leveraging a specific scaffold (citation graph) which allows for creating highly precise question. It is unclear to me why then an LLM couldn't use a similar scaffold (navigate the citation graph) at test time and obtain similar answer quality than when the dataset was created in the first place.
W4. I don't believe this draft is clear enough (in particular the experimental sections) to be broadly understandable and useful to the community.
W5. It seems fairly unlikely to me that models would struggle when given: (1) the summary of relevant works, (2) information about the cutoff (i.e., when information validity changes), and all the model has to do is answer 2 MCQ questions. But again, I do not understand the metrics and evaluation well enough to appreciate the findings.
W6. I unfortunately recognize all too well the signs of AI writing, and the lack of conceptual clarity that comes with it. The AI models introduce heavy, unnecessary jargon that makes comprehension so hard.
Questions for authors
-
One main question really: can you explain in simple terms what you think are the main results of your work, and in what way it is surprising and advances our understanding of AI systems. If at all possible, please do not use AI to generate a highly complex and jargon heavy answer, and instead try to give us the essence of what the findings are, since you are the author and you have a high-level understanding of this work which we lack.
-
If LLMs were used to create the dataset in the first place (i.e., they found the contradictions that occur in a string of papers that cite each other), what is your intuition for why they fail when provided all the evidence (see W5)?
3 comments
All means above weight the twelve open models equally; GPT results are reported separately. The tables contain individual model results rather than unexplained aggregate graphics. Codex belongs to the question-only main experiment and was not run in these supplied-summary ablations. Its absence from the ablation tables is not an omitted main-experiment result.
- Why does GPT still lose points when information is supplied? (W5; Q2)
The reviewer's intuition that this information should help is supported by the results. GPT-Sol Web's pair accuracy rises from 35.0% with the question alone to 46.5% with claim summaries and 77.0% with claims and source-history information. GPT-Luna Web rises from 27.5% to 40.5% to 68.0%.
It is important to distinguish incorrect choices from missing valid answers. In the claims-plus-source-history condition, the pair outcomes are:
| Original system | Both answers correct | Two valid choices, at least one wrong | No valid choice recorded at one or both dates |
|---|---|---|---|
| GPT-Sol Web | 154/200 (77.0%) | 14/200 (7.0%) | 32/200 (16.0%) |
| GPT-Luna Web | 136/200 (68.0%) | 17/200 (8.5%) | 47/200 (23.5%) |
Both failure types count against the published score, but an absent valid answer does not establish that the model misunderstood or ignored a date. Conditional on two valid choices, the pair success rates are 91.7% and 88.9%. These conditional rates do not replace the full-benchmark scores; they clarify why the headline scores alone would overstate a reasoning problem.
There are also original wrong choices despite returned supporting material. In a GPT-Sol Web question about identifying lockdown effects, a search candidate returned the 8 June 2020 primary study with text supporting the benchmark's early option. GPT nevertheless selected the opposing option for the 15 June cutoff. This establishes that relevant supporting content was returned before the wrong choice relative to the reference; it does not establish that the model read and deliberately rejected it. The separate diagnostic review below examines source use more directly.
- What does the new error review establish? (W2, W5; Q2)
The response analysis is intended to distinguish where an answer breaks down, not to infer private reasoning from a label. In particular, a broad heatmap category such as “evidence use” does not tell us whether supporting material was absent or was returned but not adopted. The source-level review tests that distinction directly.
We conducted a separate logged Codex-Sol diagnostic run(verified by human), identified in the run metadata as gpt-5.6-sol.
| Finding among the residual disagreements | Number | Share |
|---|---|---|
| Content supporting the historical reference option was returned, but another option was selected | 50.0% | |
| Qualifying support was not recovered: relevant material but no eligible historical version (7), or no supporting passage observed (2) | 30.0% | |
| Source/scope questions (5) or insufficient returned material (1) prevent a definitive decision | 20.0% |
Found but not adopted. Fourteen of the 15 records explicitly explain non-adoption; one selects competing evidence without explicitly rejecting the target passage. For example, a hydrated-ion question returned the original formula with intercept 7.6, but the answer expressly preferred the later correction to 7.2, published after the cutoff. In a medical-student stress trial, the original table reported P = 0.02, but the answer used a 2023 correction to P = 0.05 for a 2016 cutoff. These are observable choices between historical and corrected reports, not merely inferred retrieval failures.
Qualifying support not recovered. In a fuel-cell question, the system retrieved the publisher's updated article, still carrying the original publication date, but did not recover the historical passage supporting the reference answer. In a neo-W/neo-Z expression question, the returned code, abstract and later article did not contain the target claim of similar expression. “Not recovered” therefore does not necessarily mean that no relevant paper was found; recovering the right historical version is a distinct problem. It also does not prove that no suitable evidence exists elsewhere.
- Why can an LLM help construct the questions but fail to answer them? (W3; Q2)
As the figure illustrates, an LLM can follow citation links to locate an original paper and a later correction. During question construction, we use those links to identify the relevant papers and verify what each version reported and when it became public. At test time, finding the papers still leaves the model to decide which claim the evidence supported at the requested cutoff. The same papers can be relevant to both questions even when the correct answer changes between dates.
The error analysis in point 4 illustrates this distinction. In 50% of the residual disagreements in the separate Codex-Sol diagnostic run, the search returned a passage supporting the historical reference answer, yet the system selected another option. For example, the search returned the original medical-student stress trial table reporting P = 0.02. The system nevertheless answered the 2016 question using P = 0.05 from a correction published in 2023.
Table 2 shows that adding source identities, dates, ordering and cutoff guidance to the claim summaries raised GPT-Sol Web’s pair accuracy from 46.5% to 77.0%. The effect of providing the citation graph alone remains untested.
- Changes to presentation and terminology (W1, W2, W4, W6)
The revision defines the cutoff task and scoring before presenting results, explains the workflow names once, and separates the question-only main experiment from the supplied-information ablations. Each ablation states its inputs, comparison and interpretation. The response analysis uses the same definitions of early/late and workflow groups, and distinguishes observable response patterns from claims about internal causes. We agree that these explanations belong in the main text, not only in captions.
Response to Reviewer vEJT
Response to the reviewer
Thank you for explaining where the draft becomes difficult to follow. We agree that the paper should explain what each experiment changes and what its results establish, rather than ask readers to reconstruct this from captions. Below, we clarify the main finding, distinguish the three ablations, and provide a new source-level examination of incorrect answers. We keep the original main-experiment scores separate from the diagnostic reruns.
The task is to answer a question using evidence available by a specified date. Each pair contains the same question and options at two different cutoffs, answered in independent calls. “Early” and “late” refer to these dates. Early and late accuracy score the individual answers; pair accuracy gives credit only when both are correct. Missing or invalid answers receive no credit, with the denominator fixed at 200 pairs.
The main observation is that answering the later question correctly does not guarantee recovering the earlier answer. In the original question-only experiment, GPT-Sol Web achieved 77.5% late accuracy but 35.0% pair accuracy. Thus, 85 of its 155 late-correct cases did not also have a correct early answer. This is not simply a comparison of two metrics with different difficulty: it identifies the same cases in which success at the later date coexists with failure at the earlier date.
The results also show that providing organized source information helps substantially, sometimes bringing pair accuracy above 90%. Our conclusion is therefore not that models generally cannot understand scientific evidence. The contribution is to distinguish recognizing a claim from selecting the claim supported by the eligible historical record, and to test which supplied information helps with that distinction.
The workflow names describe who controls the research procedure, not model intelligence. Fixed follows a prescribed answering procedure; Orchestrated uses controller-planned tasks; Web allows GPT to plan searches within a configured workflow; Delegated gives the Codex system control of its research loop. Low, Medium and High are only shorthand groups for these workflows: Fixed; Orchestrated/Web; and Delegated, respectively. They are not a second experimental factor. Research-enabled workflows can search, whereas the question-only Fixed condition relies on model knowledge. Comparisons between complete systems are not controlled estimates of the effect of autonomy alone.
We have split the large results table into three model-level ablation tables. Their inputs and interpretation are now explained in the text:
(a) Source information, Table 2. The cutoff questions are retained while the supplied information changes from the question alone, to relevant claim summaries, to claims with source identities, public dates, ordering and cutoff guidance. Across the twelve open models, mean pair accuracy is 7.17%, 15.71% and 61.96% for Fixed, and 20.38%, 36.71% and 63.71% for Orchestrated. Every open model improves when source-history information is added to the summaries. Qwen3.8-27B reaches 94.0% and 96.5%, respectively. The main result is that access to claim content helps, but connecting that content to its source history helps much more. Because identities, dates, ordering and guidance are added together, this does not isolate a causal effect of dates alone.
(b) Evidence-use guidance, Table 3. Each cutoff question receives a summary selected to support its own reference answer. Standard and Guided use the same summary for that question; Guided additionally directs the system to use it while preserving the temporal restrictions. Mean pair accuracy changes from 64.17% to 72.29% for Fixed, with 11 of 12 models improving, but from 74.58% to 73.54% for Orchestrated, with only two improving. Thus, explicit evidence-use guidance helps the fixed procedure but is not a universal improvement. This is an answer-supporting diagnostic condition, not a test of finding the source unaided. The early and late questions receive their respective supporting summaries, not one identical summary across both dates.
(c) Claim recognition, Table 4. We supply the later answer's supporting summary and ask which claim it supports, removing the chronological-selection requirement. This measures single-question accuracy, not pair accuracy: the means are 83.38% for Fixed and 84.33% for Orchestrated. We do not treat the difference between these scores and pair accuracy as a temporal effect. Instead, matched-case analysis shows that 10.29% and 10.67% of all model-pair cases, respectively, have correct answers on both the undated and dated later question but not on the guided early question. Later-claim recognition therefore does not fully account for performance on the historical pair.