acceptodds
Under review as a conference paper at ICLR 2027

WildTrace: Benchmarking Natural Evidence Trails in Long-Context Reasoning

Abstract

Answering complex questions over long documents frequently requires integrating evidence that the source itself disperses naturally across distant passages. In an incident report, the operating condition, design flaw, and missed safety check that jointly explain a disaster may appear dozens of sections apart; in a novel, a character's true motive may surface only through scenes far removed from the moment it becomes relevant. This source-internal evidence integration is central to real-world long-document analysis, yet existing benchmarks largely sidestep it. Needle probes, planted facts, and reverse-engineered multi-hop chains embed evidence that may differ from the host text in distribution, placement, or register, making it unclear whether strong performance reflects genuine source reasoning or distributional artifacts. We introduce WildTrace, a benchmark of 481 tasks over 214 naturally occurring long-form sources such as technical incident reports and lesser-known literary narratives, where all evidence trails arise from the document's own causal, temporal, and narrative logic. Drawing on Pearl's causal hierarchy and prior multi-hop reasoning typologies, we define seven source-internal evidence geometries that characterize the distinct relational demands of analytical reading in long documents. A source-first construction pipeline mines candidate trails from document structure before writing questions; each item then undergoes multi-stage validation covering clue necessity, answer groundedness, rubric fidelity, contamination resistance and answerability. Across 18 frontier systems evaluated under evidence-withheld conditions, the strongest reaches 75.3%, with pronounced weaknesses on reasoning-intensive geometries such as counterfactual branching and causal attribution. As models are increasingly entrusted with real-world high-stakes analytical tasks, this gap between accessing information and reasoning over naturally dispersed evidence emerges as a defining challenge for the next stage of long-context research.

Primary area:
datasets and benchmarks
Keywords:
benchmark, long context, multi-hop reasoning, llm

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_g9oMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper builds a synthetic, multilingual long-context QA benchmark based on public documents. The authors perform human annotations to ensure the quality of the datasets, but It's unclear why they want to build this new dataset and they why they pick the particular baselines for benchmarking, and what do the results show and what are the insights we can get from this work.

Critical strengths

The authors at least did some human checking to ensure the quality of the dataset is acceptable.

Critical weaknesses

The quality of this work is worrisome, and it is clearly not ready for publication.

The writing is very unclear. Why this dataset is needed despite so many long context datasets, most of which also emphasize that evidences are scattered across different places?

The dataset is synthetically built, which makes it even more confusing. We already have so many synthetic data generation pipeline used during LLM training and we already have many long context benchmarks that are naturally occurring (meaning that they don't need human crafted labels). Why do we need one more which is neither realistic nor challenging?

The experiments are clearly subpar to a top-tier conference like ICLR. Not many results are reported, and the benchmarked methods aren't justified. There are many long context methods. Why aren't they tested on this benchmark?

Questions for authors

Is this a multilingual dataset? Lines 261-262 are very confusing

1 comment
AuthorsOpenReview ↗

rebuttal to reviewer g9oM

Thank you for raising the contribution and experimental scope. WildTrace addresses a concrete evaluation need: answering questions from naturally occurring long sources while discovering the supporting passages and preserving the relationships that make them relevant. We believe the source construction and reported analyses provide a stronger basis for this contribution than the review's characterization suggests.

Why this benchmark is useful

WildTrace combines source-first evidence-trail induction, full-document input, hidden supporting evidence, and relation-level evaluation. Candidate trails are identified from a source's existing causal, temporal, and narrative structure before the question and rubric are written. The evaluated model must recover the supporting evidence and the dependency that warrants its answer. Sections 2.2–2.5 describe this boundary, and Appendix Table 2 explicitly positions it alongside prior raw-source benchmarks, including NarrativeQA, QuALITY, and NovelQA.

We agree that dispersed evidence is already studied. WildTrace adds an explicit account of how that evidence must be composed, together with checks for grounding, necessity, shortcuts, and answerability. This supports analysis of different relational demands within the same evaluation setting and provides an auditable resource for future long-context methods.

Natural sources and constructed annotations

The 214 source documents are naturally occurring technical reports and literary narratives. Their evidence is retained in its original context and placement. Questions, reference answers, and rubrics are authored evaluation annotations. Human annotation is fully compatible with evaluating naturally occurring evidence. As disclosed in the AI Use Statement, LLMs assist evidence-record extraction and grading, with source-support checks and human review.

The two item-level issues raised elsewhere in the discussion will be handled by excluding those items and auditing the related subsets. This improves alignment with our inclusion criteria while preserving the benchmark's source-first objective. We also acknowledge query-selection and domain-coverage limitations rather than treating natural sources as a guarantee of representative user requests.

Experiments and insights

Table 1 reports 18 systems, Figure 5 provides length and geometry profiles, Tables 13–14 examine geometry-by-length and source-family patterns, and Table 15 presents a descriptive factor analysis. These analyses examine how model behavior varies across document scales and evidence relationships. They provide concrete diagnostics beyond a single aggregate leaderboard.

The standardized direct full-document protocol establishes a common capability baseline. Retrieval, iterative reading, memory, and other inference strategies can be evaluated on the same resource, which is part of its value. We will make the baseline-selection rationale more explicit. Section 5 already identifies these additional inference strategies as complementary evaluations, and the current comparison is scoped accordingly.

Language scope

The release contains 441 tasks over English sources and 40 over Chinese sources. Section 2.3 describes Chinese literature as an initial additional source slice. This expands the evidence regimes represented by the benchmark. We will make that scope more prominent and keep source-family comparisons distinct from claims about comprehensive multilingual performance.

Licensing and bias

The Ethics Statement and release include a mixed-license notice, corpus metadata, and Responsible AI documentation. The release labels benchmark annotations and scripts CC BY 4.0 and states that sources retain their original terms. We agree that clearer source-level provenance and permissions mapping will strengthen this documentation.

Appendix Tables 9–10 also report judge-family sensitivity and expert score validation. Together with the source and geometry slices, these provide explicit checks on particular evaluation biases. We will clarify which concerns each check addresses and retain the limitations concerning source selection and language coverage.

We hope these clarifications support reconsideration of the contribution and experimental-scope concerns. The paper presents a specific evaluation target, a substantial model comparison, and several diagnostic analyses. A particular prior benchmark with the same boundary, or a concrete missing comparison, would help us address any remaining concern more precisely.

Reviewer_P6D2OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

To test whether a model, after reading a long document, can locate clues scattered across different parts of the text and synthesize them to answer questions, this paper introduces WildTrace. The authors employ a source-first construction pipeline: they extract natural evidence trails from authentic technical incident reports and literary works, then develop questions, reference answers, and scoring rubrics based on these trails. This process yielded 481 tasks derived from 214 sources, covering seven distinct evidence geometries. The authors further validated the grounding of the answers in the source text through several methods, including no-document probes, checks for the necessity of specific clues, assessments of whether the evidence provided is sufficient to answer the question, and manual review. They also verified that each question requires the integration of multiple clues to be answered correctly. The study evaluates 18 systems under both full-document and evidence-withheld settings, where models are provided with the complete document and the corresponding question. Responses were graded by three independent judges (who were not involved in the competition), with the reliability of these scores verified through expert spot-checks. The top-performing system achieved a mean rubric score of 75.3%; among the seven evidence geometries, counterfactual reasoning yielded the lowest average score across all systems.

Critical strengths
  1. The authors conducted a rigorous, multi-faceted validation of the automated scoring process, which significantly enhances the credibility of the experimental results. They employed three independent judges to score the responses, assessed inter-rater reliability, and examined whether the results remained stable if the Qwen judge was excluded. Furthermore, they utilized evidence-visible judging and expert re-evaluation of 60 responses to ensure the robustness of the scoring. These measures provide a solid foundation for using automated scoring to compare model performance and bolster the overall trustworthiness of the findings.

  2. The proposed evaluation design is highly innovative, addressing a critical challenge in long-document analysis: the ability of a model to accurately preserve the relationships between pieces of evidence when synthesizing them. By utilizing natural evidence trails and seven specific evidence geometries, WildTrace effectively assesses whether a model can not only locate relevant information but also understand how that information collectively supports an answer. This approach aligns closely with the practical requirements of long-document analysis and helps identify instances where models might overlook or misinterpret the relationships between key pieces of evidence.

Critical weaknesses

The paper’s analysis regarding the reasons why models provide incomplete or inaccurate answers remains insufficient. Figure 2 illustrates errors such as the omission of certain evidence and the conflation of mechanisms; however, these examples alone are insufficient to pinpoint exactly where the failure occurs. For instance, if a model fails to account for a critical piece of evidence, it could be because it failed to locate the relevant passage, or because it found the passage but failed to correctly grasp its relationship to other pieces of evidence.

In the paper, the authors employ an evidence-conditioned answerability check to verify whether a task can be answered given the relevant evidence, a procedure primarily used to assess task quality. The paper does not, however, report results comparing the same system’s performance on the same task using full-document input versus evidence-conditioned input. Therefore, the ability to distinguish between difficulties in evidence retrieval and difficulties in evidence integration remains limited. It is recommended that this comparison be included to observe which errors are mitigated, and which persist, when the evidence is provided directly; this would help explain performance variations across different evidence geometries.

Questions for authors

Do the authors already possess results comparing the same system’s performance on the same task using full-document input versus evidence-conditioned input? If so, which evidence geometries show the greatest performance improvement when the relevant evidence is provided directly, and which types of errors still persist? These findings would help determine whether the model’s challenges stem primarily from the retrieval of evidence or from the understanding and integration of the relationships between pieces of evidence.

1 comment
AuthorsOpenReview ↗

rebuttal to Reviewer P6D2

Thank you for recognizing the value of the evidence-trail setting and the scoring validation. We agree that your proposed comparison would strengthen the explanation of model failures.

WildTrace's primary contribution is an end-to-end evaluation in which models discover supporting passages within a full source and integrate their relationships. The existing results establish performance profiles for that joint task. Supplying gold evidence would provide a complementary diagnostic by reducing the burden of finding support.

D8 verifies evidence-conditioned answerability during construction, and the evidence-visible judging study checks the scorer. We will make these roles clearer. The paper does not report a matched model-side comparison, so we cannot yet provide geometry-specific gains from supplying evidence.

A useful paired comparison would keep the model and generation settings fixed, supply raw evidence windows in source order, and withhold reference answers, rubrics, and relation labels. It would show which errors diminish when evidence is accessible and which persist. We will clarify that the current examples describe observable answer failures and that the geometry profiles remain informative descriptions of end-to-end task difficulty. We appreciate this concrete direction for extending the benchmark's diagnostic value.

Reviewer_cXBFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark to evaluate the capabilities of large language models to process information from long documents, understand how different pieces of evidence relate, and use them to answer questions. The authors note that existing benchmarks do not fully capture this ability because they often rely on artificially inserted information or preselected evidence. They introduce a source-first construction process and multiple validation steps to develop the benchmark. WildTrace includes 481 questions from 214 documents and covers seven reasoning types. They evaluate 18 language models using rubric-based scoring by three LLM judges, also with additional human validation. The results show that even the best-performing model, Gemini 3.1 Pro, achieves 75.3% average rubric credit, suggesting that there is still considerable room for improvement, in particular for more challenging reasoning tasks. The authors conclusion is that evaluating long-context reasoning should go beyond measuring how much text a model can process or whether it can retrieve specific facts. It should also evaluate whether a model can identify relevant evidence, understand the relationships between different pieces of information, and produce answers that are properly supported by the source document.

Critical strengths

The authors have made a substantial effort to validate the benchmark through many checks, including human review and the judge agreement analysis. They have evaluated 18 models across different evidence. The source first construction process is also a valuable contribution.

Critical weaknesses
  • The paper includes evidence-conditioned answerability checks during construction (D8, Table 8), but the experiments do not clearly distinguish between failing to find evidence and failing to reason about evidence that has already been found. It is difficult to determine which capability is primarily responsible for the observed performance gaps. It would help to compare model performance when the relevant evidence is provided directly versus when the model must find it in the full document. I also suggest reporting how many tasks and source documents each model can process in full and comparing models on a common set of tasks whose documents fit within their context windows.

  • The paper reports that some reasoning geometries are more difficult for models than others, Fig 5. The authors consider several factors, including document length, source type, and structural metadata (Table 15), but I am not sure whether the differences in performance can be attributed mainly to the type of reasoning required. For example, questions from different geometries might differ in how much guidance they provide, how many pieces of evidence need to be combined, or how long and detailed the required answers are.

Questions for authors
  • The paper states that the rubrics reward answer-critical relationships. What happens if a model identifies all the relevant facts but explains the causal relationship between them incorrectly? Could it still receive a relatively high score, and how does the scoring distinguish this response from one that explains the relationship correctly?
1 comment
AuthorsOpenReview ↗

rebuttal to reviewer cXBF

Thank you for the constructive suggestions and for recognizing the value of the source-first construction and validation. Your comments help sharpen the interpretation of WildTrace's results.

Evidence discovery and integration

The primary benchmark measures end-to-end success on a practical input boundary: the source and question are available, while the supporting passages must be discovered. This joint challenge is useful even when an individual failure cannot be uniquely attributed to locating evidence or integrating it. The existing length and geometry analyses describe performance within this setting.

Your proposed gold-evidence comparison would add a valuable diagnostic by reducing the evidence-discovery burden. D8 verifies answerability during construction, and Table 11 audits the judges. We will clarify their roles and distinguish them from the model-side comparison, which is not reported in the current paper.

Coverage and geometry profiles

The harness already distinguishes valid-response quality from coverage-sensitive All481 performance. We will make the denominators and context-limit exclusions more prominent. Shared-feasible comparisons would further clarify cross-model differences.

Table 13 examines geometries within length tiers, and Table 15 includes additional descriptive factors. These provide useful task profiles for identifying difficult combinations. We agree that question guidance, clue count, and required answer detail can also contribute, and will qualify the interpretation in Section 3.3 accordingly.

Facts and relations in the rubric

Rubrics give credit for answer-critical facts and relationships. An incorrect causal bridge should lose credit for criteria requiring that relation, while correct factual content can retain partial credit. The resulting total depends on the item's weights. We will make this scoring interpretation explicit so that partial rubric credit is distinguished from complete recovery of the required explanation.

These clarifications preserve the benchmark's end-to-end evaluation objective and make the diagnostic scope of its results more precise. We appreciate the suggested direction for extending that analysis.

Reviewer_MXCCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WildTrace introduces 481 long-document tasks over 214 sources, organized into seven evidence geometries. Its central proposal is to construct questions around evidence relationships already present in source documents. The construction pipeline combines source-derived trails, necessity and answerability checks, and human review. 18 models are evaluated using a three-judge rubric panel, with the strongest model achieving 75.3% mean rubric credit.

The intended contribution is a benchmark of reasoning over naturally dispersed evidence. My main concern is whether the released tasks and scoring procedure establish that interpretation of the results.

Critical strengths

The benchmark addresses a meaningful evaluation problem. Recovering relationships among naturally dispersed passages is relevant to document analysis, and the source-first approach provides a useful basis for studying it. The inclusion of technical reports and English and Chinese literary sources broadens the evaluation beyond a single document style.

The authors make a substantial effort to validate their benchmark. The necessity and answerability checks, geometry relabeling study, expert scoring audit, and evidence-visible judging comparison are valuable.

Critical weaknesses
  1. The released scoring procedure changes the task being evaluated, and the consequences are not established.

I was particularly concerned by build_judge_prompt (https://huggingface.co/datasets/nmrbench/nmr-bench/blob/e2851c5b186d44d15600241faba7716d8a8ae4cc/eval/run_judge.py): it retains only the first 600 characters of the question, 260 characters of each rubric criterion, and 5,000 characters of the response. These limits also appear in the released evaluation protocol. Counting the released data, 90 of 481 questions exceed the question limit, and 231 of 1,828 rubric criteria exceed the criterion limit, affecting 122 tasks.

The truncation can remove the substance of a criterion. For example, in wildtrace-0302, the fifth criterion loses the latter part of the required evidential chain and its explicit condition for withholding credit, while retaining its full point weight. Response truncation creates the opposite risk: a correct explanation appearing later in an answer becomes invisible to the judge. The generation prompt explicitly encourages detailed answers, making this mismatch particularly difficult to justify. The actual frequency and score impact of response truncation remain unreported.

High agreement among three judges does not resolve this problem when all three receive the same incomplete inputs. I cannot tell how much of the reported performance gap reflects model errors and how much reflects the evaluator’s altered view of the task.

The aggregation causes a further problem: tasks exceeding a model’s configured context cap are not submitted and receive zero. This measures coverage as well as answer quality. Such zeros cannot themselves support conclusions about failed evidence integration. Coverage, attempted-task quality, and performance on a shared feasible subset need to be separated. Released evaluation protocol

  1. The released examples do not consistently establish the reasoning construct that motivates the benchmark.

The paper’s central distinction depends on necessary relationships among evidence fragments. Natural provenance alone does not establish those relationships.

In wildtrace-0029, an intersection question asks for the person matching several attributes, including possession of a printing press. One released clue explicitly identifies Friedrich Bremer as possessing it. I was puzzled by how this item passed the single-clue sufficiency check, since that clue already identifies the requested entity. Requiring quotations for the remaining attributes to obtain full rubric credit establishes a requirement for corroboration, but does not establish that multiple clues are necessary to identify the answer. Is the intended construct answer discovery or exhaustive evidence verification? The distinction matters to the claimed contribution.

wildtrace-0050 raises a more serious grounding concern. Its counterfactual rubric requires the conclusion that, without Isabella’s encouragement of Captain Tilney, her later letter would not describe his persistent attentions. The cited observations do not establish that causal necessity. The source also explicitly characterizes the letter as deceptive and contradictory. I do not see what warrants treating its contents as a determinate consequence of the proposed intervention. A model questioning that inference could demonstrate appropriate reasoning while losing rubric credit. Released task data

I haven't gone through all the examples, but just these few have already raised so many questions for me. I can't help but doubt the entire design of the theory and the experiments. These examples do not establish the prevalence of invalid items. They do establish that the extensive validation procedure has not eliminated failures directly relevant to the benchmark’s central claim. Geometry-label agreement and agreement when applying the same rubric do not independently demonstrate that the underlying inference is necessary or warranted.

Questions for authors
  • Scoring fidelity. Were the reported leaderboard scores generated with the released 600/260/5,000-character truncation rules? If so, how do scores, rankings, and geometry-level conclusions change when judges receive complete questions, criteria, and answers? Please also report the proportion of actual model responses exceeding the response limit.

  • Multi-hop necessity. For wildtrace-0029, what exactly fails when only the printing-press clue is supplied? Does D7 test recovery of the requested entity, satisfaction of every rubric fact, or another criterion? Please provide the item’s validation record and clarify how the procedure distinguishes necessary inference from additional corroboration.

  • Counterfactual validity. What source evidence justifies the required counterfactual conclusion in wildtrace-0050, particularly given the unreliability of Isabella’s letter? Can a response explicitly identifying the counterfactual as underdetermined receive full credit? How frequently does independent review find similarly unsupported necessity claims in this category?

  • Interpretation of model failures. Do the length and geometry findings persist after separating out-of-context exclusions from attempted tasks and comparing models on the same feasible items?

  • Reproducibility Concerns. Appendix A.6 states that averaged and per-judge score matrices are released. I could not locate them in the inspected public revision.

3 comments
AuthorsOpenReview ↗

rebuttal to reviewer MXCC

Thank you for examining the release in detail. Your examples help us tighten the implementation of WildTrace's central objective: evaluating how models discover and integrate evidence relations already present in long sources. We will remove the two flagged items from the revised release and audit the corresponding subsets. We also clarify below the limited aggregate effect of these two removals and the separate scoring issue.

The two flagged items and their impact

For wildtrace-0029, the question requires support for all three attributes, and the two rubric criteria cover different passages. This is a meaningful corroboration task. We agree, however, that the printing-press clue already identifies the entity, so the item does not meet our stricter criterion for multi-clue answer discovery. We will exclude it from the revised strict set and make this distinction explicit in the necessity checks.

For wildtrace-0050, the source supports Isabella's public behavior, but does not establish that removing her encouragement would necessarily change Captain Tilney's persistence or the contents of her unreliable letter. We will exclude this item and audit the remaining counterfactual items for unsupported necessity claims. A response that correctly identifies an underdetermined consequence should be rewarded for that reasoning.

These two identified items represent 2 of 481 tasks, approximately 0.42% of the release. Their numerical influence can be bounded without assuming their individual scores. Since each task score lies in [0, 1], removing these two items alone changes any model's aggregate mean by at most 0.42 percentage points, provided the retained scores are fixed. For the main L0–L6 geometry analysis, removing one intersection item from 69 and one counterfactual item from 67 can change their mean-score gap by less than 2.95 percentage points. The reported gap is approximately 21 points, so these two examples alone cannot account for that broad pattern. Exact model ordering and other close comparisons should still be checked on the revised set.

The source-first construction, full-document evaluation boundary, and relation-level organization continue to provide the same research contribution. We will version the revised task set and audit related items against the stated inclusion criteria. The bounds above concern the two identified removals, without assuming that the broader audit will find no further issues.

Complete inputs for scoring

We verified the prompt limits in the cited release. The 600-character question limit and 260-character criterion limit clip 90 questions and 231 criteria, respectively, with question or criterion text clipped in 182 tasks. Your wildtrace-0302 example illustrates why retaining complete criteria matters. We will remove these character limits from the judge prompt and retain complete questions, criteria, and responses.

The three-judge panel, expert scoring audit, and evidence-visible judging check provide useful support for rubric-based grading. Complete-input score sensitivity is an additional check needed to address this implementation issue. The original generation and judging logs are absent from the public release, so we cannot yet report the frequency of response clipping or the resulting score changes. We will distinguish the item-removal bounds above from the empirical effect of correcting the judge inputs, which requires rescoring.

Context coverage and interpretation

The released code submits the full source for eligible tasks. Documents above an endpoint's configured cap are recorded as out_of_context_scope and skipped. The harness already separates scored_overall, measuring valid-response quality, from all_tasks_overall, measuring coverage-sensitive performance. The latter captures the practical ability to answer requests spanning the release's document lengths.

We agree that a skipped item is evidence of an endpoint coverage limit. It should not be used to diagnose an attempted reasoning failure. We will make this interpretation prominent and distinguish feasible-task counts, attempted-task quality, and shared-subset comparisons in reporting. This makes the benchmark's existing coverage and answer-quality dimensions easier to interpret.

AuthorsOpenReview ↗

Rebuttal to reviewer MXCC

Release completeness

Thank you for pointing out that the cited snapshot omits the score matrices described in Appendix A.6. The package already exposes the tasks, clues, rubrics, corpus, evaluation code, and explicit scoring denominators, which enabled this detailed inspection. We will correct the release description to match the available artifacts. Restoring the historical matrices and complete scoring provenance remains an outstanding reproducibility task.

Overall, the flagged examples motivate targeted data and implementation refinements. The two-item sensitivity bounds support the stability of the broad aggregate pattern under those removals, while the complete-input scoring check will establish the effect of the separate prompt correction. We hope this distinction helps assess the benchmark's contribution alongside the concrete improvements prompted by your review.

Thank you for acknowledging the problematic items and clarifying the coverage issue. The two-item removal bounds do not address the broader scoring problem affecting 182 tasks, and the impact of response clipping remains unknown. I would need complete-input rescoring and supporting score records to assess whether the reported findings hold.

I therefore maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.