- The released scoring procedure changes the task being evaluated, and the consequences are not established.
I was particularly concerned by build_judge_prompt (https://huggingface.co/datasets/nmrbench/nmr-bench/blob/e2851c5b186d44d15600241faba7716d8a8ae4cc/eval/run_judge.py): it retains only the first 600 characters of the question, 260 characters of each rubric criterion, and 5,000 characters of the response. These limits also appear in the released evaluation protocol. Counting the released data, 90 of 481 questions exceed the question limit, and 231 of 1,828 rubric criteria exceed the criterion limit, affecting 122 tasks.
The truncation can remove the substance of a criterion. For example, in wildtrace-0302, the fifth criterion loses the latter part of the required evidential chain and its explicit condition for withholding credit, while retaining its full point weight. Response truncation creates the opposite risk: a correct explanation appearing later in an answer becomes invisible to the judge. The generation prompt explicitly encourages detailed answers, making this mismatch particularly difficult to justify. The actual frequency and score impact of response truncation remain unreported.
High agreement among three judges does not resolve this problem when all three receive the same incomplete inputs. I cannot tell how much of the reported performance gap reflects model errors and how much reflects the evaluator’s altered view of the task.
The aggregation causes a further problem: tasks exceeding a model’s configured context cap are not submitted and receive zero. This measures coverage as well as answer quality. Such zeros cannot themselves support conclusions about failed evidence integration. Coverage, attempted-task quality, and performance on a shared feasible subset need to be separated. Released evaluation protocol
- The released examples do not consistently establish the reasoning construct that motivates the benchmark.
The paper’s central distinction depends on necessary relationships among evidence fragments. Natural provenance alone does not establish those relationships.
In wildtrace-0029, an intersection question asks for the person matching several attributes, including possession of a printing press. One released clue explicitly identifies Friedrich Bremer as possessing it. I was puzzled by how this item passed the single-clue sufficiency check, since that clue already identifies the requested entity. Requiring quotations for the remaining attributes to obtain full rubric credit establishes a requirement for corroboration, but does not establish that multiple clues are necessary to identify the answer. Is the intended construct answer discovery or exhaustive evidence verification? The distinction matters to the claimed contribution.
wildtrace-0050 raises a more serious grounding concern. Its counterfactual rubric requires the conclusion that, without Isabella’s encouragement of Captain Tilney, her later letter would not describe his persistent attentions. The cited observations do not establish that causal necessity. The source also explicitly characterizes the letter as deceptive and contradictory. I do not see what warrants treating its contents as a determinate consequence of the proposed intervention. A model questioning that inference could demonstrate appropriate reasoning while losing rubric credit. Released task data
I haven't gone through all the examples, but just these few have already raised so many questions for me. I can't help but doubt the entire design of the theory and the experiments. These examples do not establish the prevalence of invalid items. They do establish that the extensive validation procedure has not eliminated failures directly relevant to the benchmark’s central claim. Geometry-label agreement and agreement when applying the same rubric do not independently demonstrate that the underlying inference is necessary or warranted.
rebuttal to reviewer g9oM
Thank you for raising the contribution and experimental scope. WildTrace addresses a concrete evaluation need: answering questions from naturally occurring long sources while discovering the supporting passages and preserving the relationships that make them relevant. We believe the source construction and reported analyses provide a stronger basis for this contribution than the review's characterization suggests.
Why this benchmark is useful
WildTrace combines source-first evidence-trail induction, full-document input, hidden supporting evidence, and relation-level evaluation. Candidate trails are identified from a source's existing causal, temporal, and narrative structure before the question and rubric are written. The evaluated model must recover the supporting evidence and the dependency that warrants its answer. Sections 2.2–2.5 describe this boundary, and Appendix Table 2 explicitly positions it alongside prior raw-source benchmarks, including NarrativeQA, QuALITY, and NovelQA.
We agree that dispersed evidence is already studied. WildTrace adds an explicit account of how that evidence must be composed, together with checks for grounding, necessity, shortcuts, and answerability. This supports analysis of different relational demands within the same evaluation setting and provides an auditable resource for future long-context methods.
Natural sources and constructed annotations
The 214 source documents are naturally occurring technical reports and literary narratives. Their evidence is retained in its original context and placement. Questions, reference answers, and rubrics are authored evaluation annotations. Human annotation is fully compatible with evaluating naturally occurring evidence. As disclosed in the AI Use Statement, LLMs assist evidence-record extraction and grading, with source-support checks and human review.
The two item-level issues raised elsewhere in the discussion will be handled by excluding those items and auditing the related subsets. This improves alignment with our inclusion criteria while preserving the benchmark's source-first objective. We also acknowledge query-selection and domain-coverage limitations rather than treating natural sources as a guarantee of representative user requests.
Experiments and insights
Table 1 reports 18 systems, Figure 5 provides length and geometry profiles, Tables 13–14 examine geometry-by-length and source-family patterns, and Table 15 presents a descriptive factor analysis. These analyses examine how model behavior varies across document scales and evidence relationships. They provide concrete diagnostics beyond a single aggregate leaderboard.
The standardized direct full-document protocol establishes a common capability baseline. Retrieval, iterative reading, memory, and other inference strategies can be evaluated on the same resource, which is part of its value. We will make the baseline-selection rationale more explicit. Section 5 already identifies these additional inference strategies as complementary evaluations, and the current comparison is scoped accordingly.
Language scope
The release contains 441 tasks over English sources and 40 over Chinese sources. Section 2.3 describes Chinese literature as an initial additional source slice. This expands the evidence regimes represented by the benchmark. We will make that scope more prominent and keep source-family comparisons distinct from claims about comprehensive multilingual performance.
Licensing and bias
The Ethics Statement and release include a mixed-license notice, corpus metadata, and Responsible AI documentation. The release labels benchmark annotations and scripts CC BY 4.0 and states that sources retain their original terms. We agree that clearer source-level provenance and permissions mapping will strengthen this documentation.
Appendix Tables 9–10 also report judge-family sensitivity and expert score validation. Together with the source and geometry slices, these provide explicit checks on particular evaluation biases. We will clarify which concerns each check addresses and retain the limitations concerning source selection and language coverage.
We hope these clarifications support reconsideration of the contribution and experimental-scope concerns. The paper presents a specific evaluation target, a substantial model comparison, and several diagnostic analyses. A particular prior benchmark with the same boundary, or a concrete missing comparison, would help us address any remaining concern more precisely.