- Rating
- 4
- Confidence
- 3(+1)
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
- identifies 2 failures: unbounded exploration, evidence-state drift
- Evidence–Conclusion: Coevolution and Evidence Lifecycle Management: model-agnostic harness: interactions between evedence and conclusions
Critical strengths
- propose model agnostic harness, and evaluates its against huge corpus
- idea is clean and sensible: keep a source-linked evidence board, and let open questions drive the search
- tested on OfficeQA Pro
- more efficient on harder V2 tasks
Critical weaknesses
- a baseline would be to give these LLM models with the same tools and see how that performs (whatif gains coming from tools rather than notebook structure)
- Claude Code and Codex are tuned for Claude and GPT models. instead running them on GLM and Qwen likely handicaps them (missing frontier model results) which shrinks the claim of model agnostic with just mid sized models being tested
- there needs to analysis on why LongHarness with glm scores higher on the "harder" V2 > V1, while Qwen drops instead. (discuss deeper on why)
- token use, cost and other details should be added
- how you tackle data contamination as facts could be memorized
Questions for authors
3 comments
Response to Reviewer m1Zc [Part2]
Q3: Why does LongHarness perform better on the harder OfficeQA Pro V2 with GLM-5.3-Flash, but worse with Qwen-3.8-Flash?
A3: Thank you for this question. We further analyze interaction trajectories to understand why GLM-5.3-Flash improves slightly on OfficeQA V2 while Qwen-3.8-Flash declines. The difference is associated with how the two backbones use the harness's persistent evidence states.
The gap is concentrated in tasks requiring integration across multiple documents. Questions requiring at least 5 gold documents account for only 3% of V1 but 58% of V2. Among the 38 V2 questions requiring at most 4 documents, GLM-5.3-Flash and Qwen-3.8-Flash achieve 60.5% and 57.9%, respectively. Among the 52 questions requiring at least 5 documents, they achieve 61.5% and 46.2%, widening the gap to 15.3 percentage points. On these 52 questions, both models have a median retrieval count of 12.5 and cover 8 documents, but GLM-5.3-Flash records 145 numerical values per question in the Evidence Bank, compared with 96.5 for Qwen-3.8-Flash. As document requirements increase from at most 4 to at least 5, the former's number of recorded values per read document remains stable, changing from 18.5 to 18.2, whereas the latter's falls from 14.7 to 10.5. This suggests that GLM-5.3-Flash continues to convert intermediate results into persistent evidence that can be reused across turns as document requirements grow. Qwen-3.8-Flash instead records less evidence per document and relies more on the interaction context to retain intermediate results, making it harder to accumulate and reuse information reliably across multiple documents.
We therefore interpret GLM-5.3-Flash as adapting more actively to the harness: it makes fuller use of persistent evidence states to support subsequent reasoning, whereas Qwen-3.8-Flash more often reuses intermediate results directly from context. As V2 substantially increases the demand for multi-document integration, the latter approach becomes less reliable at maintaining and reusing information, which we view as an important contributor to Qwen-3.8-Flash's decline. In future work, we will explore incorporating persistent-state use into post-training objectives so that different backbones can benefit more consistently from the harness.
Q4: Should token usage, costs, and other computational details be reported?
A4: Thank you for this suggestion. We have added mean cumulative input, output, and total tokens per question for each method, together with an analysis of structured-state overhead, in Appendix C.4. In LongHarness requests, the Evidence State occupies approximately 748 tokens and the Conclusion State approximately 195 tokens per turn, totaling approximately 944 tokens per turn. Repeated injection across requests accounts for approximately 18,880 tokens per question, or 1.5% of cumulative input tokens. Structured states therefore account for only a small fraction of cumulative context overhead. The table reports mean token consumption per question for each harness on OfficeQA V1 with GLM-5.3-Flash as the base model.
| Method | Exact (%) | Mean cumulative input tokens | Mean cumulative output tokens | Mean cumulative total tokens |
|---|---|---|---|---|
| LongHarness | 60.2 | 1,296,944 | 49,902 | 1,346,845 |
| LongHarness (Last 5-Interactions) | 58.7 | 607,117 | 54,987 | 662,104 |
| Claude Code | 48.1 | 515,944 | 17,754 | 533,698 |
| Codex | 46.6 | 2,048,336 | 7,068 | 2,055,405 |
To examine performance at comparable computational cost, we additionally retain only the last 5 complete interactions, including chain-of-thought, ordinary responses, tool calls, and tool results. This setting uses an average of 662,104 total tokens per question, of the same order as Claude Code's 533,698 and below Codex's 2,055,405. It achieves 58.7% Exact, exceeding Claude Code's 48.1% and Codex's 46.6% by 10.6 and 12.1 percentage points, respectively. Even without the full interaction history, the model can maintain evidence and conclusions across turns through the Evidence Bank and Conclusion State and complete the reasoning task. This indicates that structured persistent states can support long-horizon reasoning with lower context cost.
Response to Reviewer m1Zc [Part3]
Q5: How do you address possible data contamination from facts memorized during pretraining?
A5: Thank you for raising this concern. We distinguish answer leakage during evaluation from potential memorization during pretraining. Regarding the former, all harnesses receive only the question and the complete corpus, without gold documents, reference answers, or source annotations. Benchmark answers and annotations are not embedded in the harness. LongHarness retains its advantage over the baseline harnesses when using the same backbone, questions, corpus, and scorer.
Regarding pretraining memorization, complex corpus-level reasoning requires more than retrieving facts. It also requires locating and aligning data across documents and then performing computations. With GLM-5.3-Flash as the backbone, LongHarness correctly answers 80 and 55 questions on OfficeQA Pro V1 and V2, respectively. Among these successful trajectories, 86% and 91% of final answers cite at least two different documents, with a median of 3. Computation occurs before submission in 98% and 95% of cases, and the final numerical value does not appear verbatim in any cited passage in 91% and 95% of cases. Even if individual values appeared in pretraining data, LongHarness still needs to locate the correct tables, years, statistical definitions, and versions among 697 or 1,435 documents and then integrate the evidence and perform the required computations. Taken together, the controlled comparisons with the same backbone and this trajectory evidence lead us to attribute the performance advantage of LongHarness primarily to effective organization of documentary evidence and cross-document reasoning, rather than answer leakage or direct recall of pretrained facts.
Response to Reviewer m1Zc [Part1]
We thank Reviewer m1Zc for the comments on tool fairness, model adaptability, computational overhead, and data contamination. We provide comparisons using the LongHarness tool set, native frontier-model comparisons, and token-usage statistics. We also analyze the interaction trajectories of GLM-5.3-Flash and Qwen-3.8-Flash and examine data-contamination concerns through the evaluation protocol and trajectory evidence.
A1: Thank you for this suggestion. To directly test gains beyond the tool set, we provide controlled experiments using the same basic tools. Claude Code and Codex are given LongHarness tools, and we include a LongHarness variant that uses only these tools without evidence–conclusion coevolution or evidence lifecycle management. This tests the contributions of state structure and management in the full method. LongHarness-tools-only denotes the variant using only the basic LongHarness tool set. All these tool-alignment experiments use GLM-5.3-Flash as the base model. The reported metric is Exact accuracy on OfficeQA V1 and V2.
LongHarness tools improve Claude Code on both V1 and V2 and improve Codex on V1, while Codex declines on V2. Tools designed for corpus-level reasoning can therefore improve the fit of general-purpose file and shell operations to the task, although gains also depend on the harness and task. A suitable tool set with a basic interaction loop already achieves performance comparable to or higher than more complex harnesses such as Codex and Claude Code, demonstrating that appropriate tools can provide effective evidence access and processing for complex corpus-level reasoning. Building on these tools, full LongHarness introduces evidence–conclusion coevolution and evidence lifecycle management, exceeding LongHarness-tools-only by 7.6 and 14.4 percentage points on V1 and V2. This reflects the additional benefit of managing evidence and conclusions across turns and supports the design of LongHarness, which strengthens long-horizon reasoning through state management on top of task-adapted tools and a simple interaction loop. We have added the tool-set comparisons in Appendix C.3 and the definition and results of LongHarness-tools-only in Section 3.3 and Table 2.
A2: Thank you for pointing out the limited coverage of native frontier models in the original experiments. Our goal is to compare harnesses rather than model capabilities. We therefore held the backbone fixed and minimized harness-specific adaptation to examine plug-and-play performance. Claude and GPT models often undergo substantial adaptation to their corresponding harnesses, so evaluation in their native frameworks may combine model-adaptation benefits with the effects of the harness itself. GLM-5.3-Flash and Qwen-3.8-Flash have less comparable harness-specific adaptation and are therefore useful for controlled harness comparisons. We nevertheless agree that coverage limited to mid-sized models is insufficient. We therefore provide native and cross-harness combinations using Claude Opus 5.5 and GPT-6.1-Sol on OfficeQA Pro V1:
LongHarness achieves 71.43% with Claude Opus 5.5 and 72.93% with GPT-6.1-Sol, exceeding the corresponding native Claude Code and Codex combinations by 6.02 and 6.76 percentage points, respectively. These results extend the controlled comparisons with GLM-5.3-Flash and Qwen-3.8-Flash to native Claude and GPT frontier models. LongHarness achieves higher accuracy across both model families and their native harness combinations, indicating that its effectiveness is not confined to a particular model family or a single adaptation setting.