acceptodds
Under review as a conference paper at ICLR 2027

Research Agents Get Lost in Evolving Evidence

Abstract

A research question may ask what was supported by the evidence available at a particular date. Scholarly corrections and revisions provide documented cases in which that answer changes. We study this requirement as temporal grounding and introduce Time-conditioned Scholarly Conflict Question Answering (TSC-QA), a benchmark of 200 question pairs across eight scientific fields. The questions in each pair differ only in their evidence cutoff and require different answers. Strict-pair accuracy requires both answers to be correct. To test this requirement across different ways of conducting research, we evaluate fixed answerers, planned workflows, and systems that control their own research loop. These include Qwen, Llama, Gemma, DeepSeek, and GPT models, as well as the complete Codex-Luna, Codex-Terra, and Codex-Sol systems. When systems must find or recall the evidence themselves, the highest pair accuracy is 35.50%, and the delegated research systems perform better at the later cutoff. Providing claims organized by source and public date improves strict-pair accuracy more than providing claim summaries alone. Models can recognize a supplied claim and answer the later question correctly while still missing the earlier answer. Response analysis connects these results to claim interpretation, source dating, and evidence use. TSC-QA tests whether research agents can recover the answer supported at each point in a changing evidence history.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vEJTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces the TSC-QA collection of 200 question pairs, associated with a corpus of scholarly documents. Each sample is a question that has two contradicting answers, depending on what time cutoff we consider (e.g., something that was believed to be true at time X1, and is refuted at a later time X2). Evaluation of models is on the capability of models to answer both questions correctly in different sessions (get the early answer for an early cutoff, and the late answer with the late cutoff).

Critical strengths
  • The work of time-dependence on scientific claims is interesting and timely, the dataset, built based on the citation graph, is an interesting procedure.
  • The example provided in the introduction is clear and compelling, this is one of the few things that is clear in the presentation of the work.
Critical weaknesses

W1. Everything from Section 4 onwards needs to be rewritten, as it is absolutely unclear what is going on. The paper goes in very rapid succession from one experiment to the next, with barely any explanation, or interpretation of the results. For example:

  • Table 1 introduces Workflow, Procedure and Group. It is unclear that this is ever used again (particularly the Group).
  • Figure 3 is a sort of cherry-picked selection of Table 2, which is the main Table of results. Mind-blowingly, the draft describes the results in Table 2 for all of 1 paragraphs (lines 308-312) before moving on. I stared at the raw table for more than 5 minutes and could not understand what the main insights are that I should take away as a reader. The authors need to explain the results; this is not a dump of data, this is a paper meant for an audience who needs to understand the main critical findings.
  • Lots of experimental details are actually in the Table captions (Source information / Standard vs. Guided Evidence use) Web vs. Fixed for GPT. It is not acceptable to have so little detail on the main experiment of the paper. The paper claims Table 2 contains all results, but the Codex models seem to be missing (unsure why), which points to one of many confusions.
  • Tables 3,4,5 (which aren't Tables and should be figures...) are somehow rehashes (?) of the main results, averaged in an unknown way. Unclear how these are generated.

W2. The paper then goes full steam ahead and does an LLM-based analysis of reasoning traces (response analysis), with a lot of new nomenclature (Early / Late) and Low Medium High. It is unclear what the value of this analysis is, as the text mostly describes values in the Table, and does not help us interpret the results.

W3. It seems like the dataset was generated with the help of AI (LLM-based) leveraging a specific scaffold (citation graph) which allows for creating highly precise question. It is unclear to me why then an LLM couldn't use a similar scaffold (navigate the citation graph) at test time and obtain similar answer quality than when the dataset was created in the first place.

W4. I don't believe this draft is clear enough (in particular the experimental sections) to be broadly understandable and useful to the community.

W5. It seems fairly unlikely to me that models would struggle when given: (1) the summary of relevant works, (2) information about the cutoff (i.e., when information validity changes), and all the model has to do is answer 2 MCQ questions. But again, I do not understand the metrics and evaluation well enough to appreciate the findings.

W6. I unfortunately recognize all too well the signs of AI writing, and the lack of conceptual clarity that comes with it. The AI models introduce heavy, unnecessary jargon that makes comprehension so hard.

Questions for authors
  1. One main question really: can you explain in simple terms what you think are the main results of your work, and in what way it is surprising and advances our understanding of AI systems. If at all possible, please do not use AI to generate a highly complex and jargon heavy answer, and instead try to give us the essence of what the findings are, since you are the author and you have a high-level understanding of this work which we lack.

  2. If LLMs were used to create the dataset in the first place (i.e., they found the contradictions that occur in a string of papers that cite each other), what is your intuition for why they fail when provided all the evidence (see W5)?

3 comments
AuthorsOpenReview ↗

Response to Reviewer vEJT

Response to the reviewer

Thank you for explaining where the draft becomes difficult to follow. We agree that the paper should explain what each experiment changes and what its results establish, rather than ask readers to reconstruct this from captions. Below, we clarify the main finding, distinguish the three ablations, and provide a new source-level examination of incorrect answers. We keep the original main-experiment scores separate from the diagnostic reruns.

  1. What is the main finding? (Q1; W1, W4, W5)

The task is to answer a question using evidence available by a specified date. Each pair contains the same question and options at two different cutoffs, answered in independent calls. “Early” and “late” refer to these dates. Early and late accuracy score the individual answers; pair accuracy gives credit only when both are correct. Missing or invalid answers receive no credit, with the denominator fixed at 200 pairs.

The main observation is that answering the later question correctly does not guarantee recovering the earlier answer. In the original question-only experiment, GPT-Sol Web achieved 77.5% late accuracy but 35.0% pair accuracy. Thus, 85 of its 155 late-correct cases did not also have a correct early answer. This is not simply a comparison of two metrics with different difficulty: it identifies the same cases in which success at the later date coexists with failure at the earlier date.

The results also show that providing organized source information helps substantially, sometimes bringing pair accuracy above 90%. Our conclusion is therefore not that models generally cannot understand scientific evidence. The contribution is to distinguish recognizing a claim from selecting the claim supported by the eligible historical record, and to test which supplied information helps with that distinction.

  1. What are the workflows and ablations? (W1, W5)

The workflow names describe who controls the research procedure, not model intelligence. Fixed follows a prescribed answering procedure; Orchestrated uses controller-planned tasks; Web allows GPT to plan searches within a configured workflow; Delegated gives the Codex system control of its research loop. Low, Medium and High are only shorthand groups for these workflows: Fixed; Orchestrated/Web; and Delegated, respectively. They are not a second experimental factor. Research-enabled workflows can search, whereas the question-only Fixed condition relies on model knowledge. Comparisons between complete systems are not controlled estimates of the effect of autonomy alone.

We have split the large results table into three model-level ablation tables. Their inputs and interpretation are now explained in the text:

(a) Source information, Table 2. The cutoff questions are retained while the supplied information changes from the question alone, to relevant claim summaries, to claims with source identities, public dates, ordering and cutoff guidance. Across the twelve open models, mean pair accuracy is 7.17%, 15.71% and 61.96% for Fixed, and 20.38%, 36.71% and 63.71% for Orchestrated. Every open model improves when source-history information is added to the summaries. Qwen3.8-27B reaches 94.0% and 96.5%, respectively. The main result is that access to claim content helps, but connecting that content to its source history helps much more. Because identities, dates, ordering and guidance are added together, this does not isolate a causal effect of dates alone.

(b) Evidence-use guidance, Table 3. Each cutoff question receives a summary selected to support its own reference answer. Standard and Guided use the same summary for that question; Guided additionally directs the system to use it while preserving the temporal restrictions. Mean pair accuracy changes from 64.17% to 72.29% for Fixed, with 11 of 12 models improving, but from 74.58% to 73.54% for Orchestrated, with only two improving. Thus, explicit evidence-use guidance helps the fixed procedure but is not a universal improvement. This is an answer-supporting diagnostic condition, not a test of finding the source unaided. The early and late questions receive their respective supporting summaries, not one identical summary across both dates.

(c) Claim recognition, Table 4. We supply the later answer's supporting summary and ask which claim it supports, removing the chronological-selection requirement. This measures single-question accuracy, not pair accuracy: the means are 83.38% for Fixed and 84.33% for Orchestrated. We do not treat the difference between these scores and pair accuracy as a temporal effect. Instead, matched-case analysis shows that 10.29% and 10.67% of all model-pair cases, respectively, have correct answers on both the undated and dated later question but not on the guided early question. Later-claim recognition therefore does not fully account for performance on the historical pair.

All means above weight the twelve open models equally; GPT results are reported separately. The tables contain individual model results rather than unexplained aggregate graphics. Codex belongs to the question-only main experiment and was not run in these supplied-summary ablations. Its absence from the ablation tables is not an omitted main-experiment result.

  1. Why does GPT still lose points when information is supplied? (W5; Q2)

The reviewer's intuition that this information should help is supported by the results. GPT-Sol Web's pair accuracy rises from 35.0% with the question alone to 46.5% with claim summaries and 77.0% with claims and source-history information. GPT-Luna Web rises from 27.5% to 40.5% to 68.0%.

It is important to distinguish incorrect choices from missing valid answers. In the claims-plus-source-history condition, the pair outcomes are:

Original systemBoth answers correctTwo valid choices, at least one wrongNo valid choice recorded at one or both dates
GPT-Sol Web154/200 (77.0%)14/200 (7.0%)32/200 (16.0%)
GPT-Luna Web136/200 (68.0%)17/200 (8.5%)47/200 (23.5%)

Both failure types count against the published score, but an absent valid answer does not establish that the model misunderstood or ignored a date. Conditional on two valid choices, the pair success rates are 91.7% and 88.9%. These conditional rates do not replace the full-benchmark scores; they clarify why the headline scores alone would overstate a reasoning problem.

There are also original wrong choices despite returned supporting material. In a GPT-Sol Web question about identifying lockdown effects, a search candidate returned the 8 June 2020 primary study with text supporting the benchmark's early option. GPT nevertheless selected the opposing option for the 15 June cutoff. This establishes that relevant supporting content was returned before the wrong choice relative to the reference; it does not establish that the model read and deliberately rejected it. The separate diagnostic review below examines source use more directly.

  1. What does the new error review establish? (W2, W5; Q2)

The response analysis is intended to distinguish where an answer breaks down, not to infer private reasoning from a label. In particular, a broad heatmap category such as “evidence use” does not tell us whether supporting material was absent or was returned but not adopted. The source-level review tests that distinction directly.

We conducted a separate logged Codex-Sol diagnostic run(verified by human), identified in the run metadata as gpt-5.6-sol.

Finding among the residual disagreementsNumberShare
Content supporting the historical reference option was returned, but another option was selected50.0%
Qualifying support was not recovered: relevant material but no eligible historical version (7), or no supporting passage observed (2)30.0%
Source/scope questions (5) or insufficient returned material (1) prevent a definitive decision20.0%

Found but not adopted. Fourteen of the 15 records explicitly explain non-adoption; one selects competing evidence without explicitly rejecting the target passage. For example, a hydrated-ion question returned the original formula with intercept 7.6, but the answer expressly preferred the later correction to 7.2, published after the cutoff. In a medical-student stress trial, the original table reported P = 0.02, but the answer used a 2023 correction to P = 0.05 for a 2016 cutoff. These are observable choices between historical and corrected reports, not merely inferred retrieval failures.

Qualifying support not recovered. In a fuel-cell question, the system retrieved the publisher's updated article, still carrying the original publication date, but did not recover the historical passage supporting the reference answer. In a neo-W/neo-Z expression question, the returned code, abstract and later article did not contain the target claim of similar expression. “Not recovered” therefore does not necessarily mean that no relevant paper was found; recovering the right historical version is a distinct problem. It also does not prove that no suitable evidence exists elsewhere.

  1. Why can an LLM help construct the questions but fail to answer them? (W3; Q2)

As the figure illustrates, an LLM can follow citation links to locate an original paper and a later correction. During question construction, we use those links to identify the relevant papers and verify what each version reported and when it became public. At test time, finding the papers still leaves the model to decide which claim the evidence supported at the requested cutoff. The same papers can be relevant to both questions even when the correct answer changes between dates.

The error analysis in point 4 illustrates this distinction. In 50% of the residual disagreements in the separate Codex-Sol diagnostic run, the search returned a passage supporting the historical reference answer, yet the system selected another option. For example, the search returned the original medical-student stress trial table reporting P = 0.02. The system nevertheless answered the 2016 question using P = 0.05 from a correction published in 2023.

Table 2 shows that adding source identities, dates, ordering and cutoff guidance to the claim summaries raised GPT-Sol Web’s pair accuracy from 46.5% to 77.0%. The effect of providing the citation graph alone remains untested.

  1. Changes to presentation and terminology (W1, W2, W4, W6)

The revision defines the cutoff task and scoring before presenting results, explains the workflow names once, and separates the question-only main experiment from the supplied-information ablations. Each ablation states its inputs, comparison and interpretation. The response analysis uses the same definitions of early/late and workflow groups, and distinguishes observable response patterns from claims about internal causes. We agree that these explanations belong in the main text, not only in captions.

Reviewer_87CYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TSC-QA, a benchmark for evaluating whether research agents can answer scientific questions under different evidence cutoffs. It contains 200 question pairs across eight scientific fields, where each pair differs only in the cutoff date and requires a different answer. Experiments across multiple LLMs and research workflows show that agents struggle to recover historical scientific conclusions. Providing source and temporal information substantially improves performance.

Critical strengths
  1. The paper addresses an important problem in scientific research, where conclusions evolve as new evidence becomes available.
  2. The paired-question design provides a clear and controlled way to evaluate temporal grounding.
  3. The experiments cover multiple model families and research workflows, with additional analyses of evidence availability and error patterns.
Critical weaknesses
  1. Limited distinction from existing benchmarks. Previous benchmarks such as EvolveBench and TempoMed-Bench have already explored temporal knowledge and evolving evidence. Although TSC-QA focuses on scholarly conflicts and paired questions, its additional evaluation value over existing benchmarks could be better demonstrated through more systematic comparisons.
  2. Confounded workflow comparisons. Although the authors provide same-model comparisons, different workflows still involve varying tools and computational budgets. In particular, the Delegated systems are not directly compared with less autonomous workflows under equivalent conditions. This makes it difficult to isolate the contribution of agent autonomy.
  3. Insufficient ablation studies. The Claims + Dates condition simultaneously introduces source identities, publication dates, source ordering, and cutoff guidance. The substantial improvement cannot be attributed to any individual component. More fine-grained ablations would help clarify which factors are most important for temporal grounding.
  4. Insufficient separation of retrieval and reasoning failures. The low Question-only accuracy may arise from agents failing to retrieve relevant historical sources rather than incorrectly reasoning about the evidence. Although the paper provides error analysis, it does not systematically distinguish these two failure modes. A controlled evaluation separating retrieval failures from reasoning failures would provide stronger insights into the limitations of research agents.
Questions for authors

see weaknesses

2 comments

Response to the second set of comments

Thank you for identifying where the paper's contribution and experimental conclusions need to be more precise. We agree that temporal answering is not new, that the workflow comparisons do not isolate autonomy, and that retrieval and answer selection must be distinguished.

  1. Distinction from EvolveBench and TempoMed-Bench

We agree that neither evolving knowledge nor historical answering is unique to our work. EvolveBench evaluates several dimensions of temporal competence using time-varying facts, including temporally misaligned external context. TempoMed-Bench studies evolving clinical guidelines, historical and current recommendations, temporal consistency, and agentic search. In particular, evaluating historical answers or adding search tools is not by itself a distinction from TempoMed-Bench. EvolveBench; TempoMed-Bench.

Our intended contribution is the combination of a different evidence domain and a matched evaluation design:

BenchmarkSource of temporal changeMain evaluation emphasis
EvolveBenchTime-varying factual attributes, principally from WikidataMultiple temporal abilities, including handling past/future context misalignment
TempoMed-BenchSuccessive clinical guideline recommendationsCurrent and historical medical knowledge, temporal consistency and agentic retrieval
TSC-QADocumented changes or opposing claims in scholarly source histories across disciplinesThe same scientific question and options at two cutoffs, scored jointly, with source-information and evidence-use controls

The prior-work descriptions follow the respective papers: EvolveBench, Section 2 and TempoMed-Bench, Sections 3–4.

The additional evaluation value is to ask whether the same system recovers both source-supported positions for a fixed question, rather than succeeding only on its later version. For example, original GPT-Sol Web achieved 77.5% late accuracy but 35.0% pair accuracy: 85 of its 155 late-correct cases lacked a correct early answer. The supplied-information controls then test which parts of this task become easier when relevant claims and their source histories are provided.

This is a complementary contribution, not a claim that prior benchmarks omit all such temporal difficulties. We have not run a matched cross-benchmark evaluation under identical models, tools and budgets, so the current evidence does not establish greater difficulty, a superior benchmark, or a failure pattern absent from existing work. A direct controlled comparison would strengthen the empirical distinction.

  1. Confounded workflow comparisons

We agree. The main experiment compares complete research configurations, not an isolated autonomy variable. Same-model comparisons reduce confounding by model identity, but still differ in procedures, tools or inference budgets. The Delegated systems differ in additional implementation details, so their scores cannot be interpreted as the causal effect of granting more autonomy.

Low, Medium and High should therefore be read as descriptive groups for workflow control: prescribed answering, bounded orchestration/search, and delegated research. They are not levels of model intelligence. The conclusion is that the tested configurations show different performance on the task, not that autonomy alone causes those differences or necessarily improves temporal reasoning. A causal comparison would require a common answer model, evidence access, tools and budget, varying only which decisions the controller or agent can make. We have not completed that comparison.

  1. Insufficient decomposition of the source-information intervention

We agree that Claims + Dates bundles several changes. Its results establish the utility of the combined source-history package: mean pair accuracy rises from 15.71% to 61.96% for Fixed and from 36.71% to 63.71% for Orchestrated relative to claim summaries. They do not establish that dates are the dominant component.

The other ablations answer narrower but different questions. Standard versus Guided holds the supporting summary fixed and changes evidence-use instructions; Evidence-Only asks for recognition of the supplied later claim without chronological selection. Neither identifies the separate contributions of identities, dates, ordering and cutoff guidance in the source-information experiment. We therefore distinguish these comparisons explicitly rather than present them as a complete component-level decomposition.

A finer experiment should hold source passages and content constant, vary those metadata/instruction components independently, and test interactions: a date may be useful only when it is correctly attached to a claim-bearing source. That experiment remains to be conducted. The present conclusion concerns the information package, not an identified individual mechanism.

  1. Separating retrieval from reasoning failures

We agree that low Question-only accuracy alone cannot establish a reasoning failure. A system may lack the relevant source, retrieve the wrong version, misapply a cutoff after finding the source, or fail to return a valid answer. These situations require different evidence and should not be collapsed into one interpretation.

Our newly logged Codex-Sol diagnostic provides a limited source-level distinction. Of the 30 residual reference-answer disagreements in the selected rerun, 15 (50.0%) had returned content supporting the historical reference option but selected another option; nine (30.0%) did not recover qualifying support, comprising seven historical-version gaps and two cases with no observed supporting passage; and six (20.0%) remained unresolved because of source/scope questions or insufficient returned material. All 30 returned valid final choices. The review considered every returned candidate, not only the final citations.

For instance, a post-cutoff correction was explicitly preferred over an already returned original result, whereas another case retrieved only a publisher's updated page despite its old publication label. These illustrate selection and version-recovery problems respectively. However, 14 of the 15 non-adoption cases also carry question-rule or scope caveats. We therefore report observable non-adoption relative to the reference, not 15 proven reasoning failures or a causal effect of prior knowledge.

This diagnostic does not fully resolve the reviewer's controlled-evaluation request. It concerns a selected subset and a new logged run, not complete original trajectories for all scored systems. Its percentages must not be extrapolated to the original benchmark errors. A stronger test would hold the answer model and evidence budget fixed, compare ordinary with version-aware retrieval, and then supply the same verified eligible passages to isolate answer selection from source discovery. Until that comparison is available, our evidence supports the narrower conclusion that both evidence recovery and subsequent source selection deserve separate evaluation.

Reviewer_SA3GOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether research agents can answer scientific questions using evidence made public by a specified time. The authors introduce TSC-QA, comprising 200 question pairs across eight fields, where only the evidence cutoff changes and paired scoring requires both date-specific answers to be correct. They compare multiple models and research workflows, finding that historical questions remain difficult and that providing source and date information substantially improves performance.

Critical strengths
  • The paper is well motivated, examining whether research agents can answer using evidence available at a specified time rather than simply providing the current conclusion.
  • The authors' paired design is straightforward, changing only the evidence cutoff, and can reveal problems overlooked when evaluating accuracy only at the later date.
  • The paper covers eight scientific fields and compares multiple models and research workflows, providing fairly extensive experimental results.
Critical weaknesses
  • The human verification reported in Appendix A.1 covers only 50 questions with GPT assistance, and the cross-disciplinary answer key still lacks sufficient independent validation.
  • The source-information experiment adds source identities, dates, ordering, and cutoff guidance simultaneously, making it unclear which component primarily accounts for the performance improvement.
Questions for authors
  • The paper should consider adding a baseline combining source-version retrieval with cutoff-date filtering to test whether it can improve the low scores of general research workflows.
  • The authors might check the error classifications using original trajectories corresponding to the current scores, distinguishing failure to retrieve evidence from failure to use retrieved evidence correctly.
  • It would be helpful to clarify why additional evidence-use guidance improves mean pair accuracy for Fixed but not Orchestrated.
2 comments

Thank you for these suggestions. They identify important distinctions between answer-key validation, the effect of supplying source information, and the diagnosis of model errors. We address each separately and distinguish completed analyses from experiments that remain to be done.

  1. Independent validation of the answer key

We agree that the reported check does not establish independent, cross-disciplinary validation of the complete answer key. Appendix A.1 describes one PhD-level reviewer checking 50 questions with GPT assistance. The reported 100% is the verification pass rate within that checked sample, not an estimate of full-dataset correctness or agreement between independent experts. GPT assistance also means this is not an unassisted human-validation result.

The source links, supporting passages and version dates make the reference answers inspectable, but inspectability does not replace independent adjudication. A stronger validation would use domain-appropriate reviewers who assess the source passages and dates independently of model predictions, document disagreements, and adjudicate ambiguous scope or chronology. That additional validation has not been completed. We therefore treat the existing exercise as an assisted spot check and do not use it to claim that all benchmark disagreements are certified model errors.

  1. What accounts for the Claims + Dates improvement?

We agree that this experiment tests an information package, not publication dates in isolation. Relative to claim summaries, it adds source identities, dates, ordering and cutoff guidance together. Mean pair accuracy increases from 15.71% to 61.96% for Fixed and from 36.71% to 63.71% for Orchestrated. Every open-model row improves, supporting the conclusion that providing organized source histories helps. These results do not identify which component contributes most.

The separate Standard-versus-Guided comparison holds the answer-supporting summary fixed for each question and changes the evidence-use instruction. It provides a narrower instruction comparison, but it does not decompose the Claims + Dates gain because the evidence preparation differs between these experiments. A component-level test would need identical source passages and candidate sets while independently varying identities, dates, ordering and guidance, including relevant interactions. We have not completed that factorial experiment and do not attribute the bundled improvement to dates alone.

  1. Source-version retrieval followed by cutoff filtering

We agree that this is an appropriate targeted baseline. It should retrieve candidate source versions, establish the public date of the claim-bearing version, exclude versions later than the question's cutoff, and pass the remaining evidence to the answerer. Filtering by the original publication date of the work is insufficient if the returned page contains a later revision.

A controlled comparison would hold the answer model, candidate pool and retrieval budget fixed while enabling or disabling version-aware filtering. It should report both eligible-support retrieval and answer accuracy conditional on retrieved support, as well as overall pair accuracy. The supplied-information experiments are not a substitute for this baseline: they provide prepared evidence rather than requiring the system to recover it. We have not yet completed the proposed baseline, so we do not claim that general-workflow results establish a limitation of a purpose-built version-aware system.

  1. Original trajectories and retrieval versus evidence-use failures

We agree that attribution of the reported scores should use trajectories from the runs that produced those scores. Recovering an original final answer or generating a retrospective explanation is not equivalent to recovering its original search returns. Where those returns are missing, we cannot reliably determine whether eligible support was retrieved.

To examine this distinction directly, we separately reran Codex-Sol on 94 individual questions selected from its original evidence-use-labelled errors, with detailed evidence logging. The new error analysis includes only the 30 rerun answers that still disagree with the frozen reference. All 30 have valid final choices. Review covered all returned candidates, rather than only final citations:

Observed outcome among the residual disagreementsCountShare
Historical target-supporting content returned, but another option selected50.0%
Qualifying support not recovered: historical-version gap (7) or no supporting passage observed (2)30.0%
Source/scope questions (5) or insufficient returned material (1) prevent a definitive decision20.0%

For example, one answer received the original P = 0.02 result but selected a post-cutoff correction to P = 0.05. Another retrieved an updated publisher page without recovering its relevant historical version. These are different observable situations. However, 14 of the 15 non-adoption cases also have question-rule or scope caveats, so their count is not a count of independently certified reasoning failures. Nor does non-adoption prove that prior knowledge overrode evidence.

These percentages describe only the selected rerun disagreements. They do not estimate failure prevalence in the original scored runs and do not replace their scores or error distribution. The new review improves the evidence available for diagnosis, but does not retrospectively supply missing original trajectories.

  1. Why does guidance help Fixed but not Orchestrated on average?

The effect is heterogeneous rather than a uniform benefit or harm. Fixed improves from 64.17% to 72.29%, with 11 of 12 models improving. Orchestrated changes from 74.58% to 73.54%, with two improvements, seven declines and three ties. Its largest declines are Llama-3.1-70B, from 35.0% to 26.0%, and Qwen3-Next-80B, from 86.5% to 79.0%; Qwen3.5-27B and Qwen3.5-122B instead improve by 4.5 and 3.5 percentage points.

One plausible explanation is that an explicit evidence-use instruction has different effects in a direct answering procedure and in a workflow that already plans and transforms the evidence before answering. In the latter, its value may depend on how the instruction reaches the final decision. The aggregate results alone do not establish that explanation, and we do not interpret the small mean decline as a statistically established harmful effect. The supported conclusion is narrower: guidance provides a consistent benefit across the tested Fixed models, but does not transfer uniformly to Orchestrated workflows. Establishing the mechanism would require matched trajectory checks or controlled interventions on the orchestration procedure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.