- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces LongHarness Bench, a benchmark evaluating language-model harnesses rather than base models alone. It comprises four synthetic long-context tasks: Constraint Solving Search, Equivalent Program Pair Search, Program Execution Tracing, and Outlier Memo Detection. These tasks require retrieval, multi-step reasoning, and adaptive evidence selection, with different strategies involving different computational costs.
Critical strengths
S1. Existing benchmarks primarily evaluate model performance, often under a fixed harness, making it difficult to assess the impact of different harness designs. This paper explicitly examines harness quality by holding the underlying model and context fixed while varying the harness configuration (Table 2). This is a useful distinction from existing benchmarks. The comparison in Table 1 also provides reasonable support for the claim that existing suites, including OOLONG, LongProc, LongBench-v2, and HELMET, do not directly address this setting. The experimental results further suggest that the proposed tasks are more challenging and better at distinguishing between harness configurations. No single configuration consistently performs best across all four tasks, and macro accuracy ranges from 0% to 68%, compared with the much narrower 60–68% range reported for LongBench-v2. The results provide a reasonable case for evaluating harness design separately from the underlying model.
S2. The tasks are generated from hidden structured specifications, with distractors introduced through controlled perturbations rather than random modifications. The authors also employ a private solver and an independent rule-based validator to recompute the ground-truth answers and verify the intended roles of required items and distractors (Appendix B, Table 5). Several task-specific design choices are particularly useful. In Constraint Solving Search, the prevalence constraints (≤10% and ≥60%) create meaningful differences in the cost of alternative search orders. In Program Execution Tracing, decoy cells are retained only when they agree on the displayed call but differ on a separate contract probe. In Equivalent Program Pair Search, true matching pairs are required to rank below at least twelve mutants under the similarity measure. These constraints make the task difficulty more convincing, particularly where different search strategies incur different costs. I also appreciate that the authors explicitly acknowledge the limitations of their ground-truth construction, particularly that program equivalence is defined operationally through testing rather than established through formal verification.
Critical weaknesses
W1. This is my main concern, as cost efficiency is central to the paper's contribution. Using the token counts in Table 4b and the rates in Table 6, I cannot reproduce several costs reported in Table 2. The discrepancies are particularly large for agentic harnesses. For example, GPT-5.6-sol with OpenCode on Constraint Solving Search costs \7.32 based on the reported tokens and rates. Prompt caching does not fully explain these differences. Since these costs underpin the accuracy–cost comparisons, the authors should clarify their accounting methodology, including cached tokens and sub-call costs, and verify the reported results. The paper should report token counts and model calls alongside dollar costs so that these comparisons can be checked.
W2. The paper does not clearly report the number of evaluation instances per task, confidence intervals, or run-to-run variability. Without uncertainty estimates, I am not convinced that the smaller accuracy differences are meaningful. For example, several comparisons in Section 4.3 differ by only a few percentage points, yet are used to support broader conclusions about differences between model families. I would like to see the sample sizes and uncertainty estimates reported explicitly. It would also help to distinguish incorrect answers from formatting failures, timeouts, and budget exhaustion, since these failure modes have different implications for evaluating harness effectiveness.
Questions for authors
Q1. What exactly is included in the per-instance cost calculation? Please distinguish uncached input, cached input, output tokens, and any sub-calls made by agentic harnesses. Were the token counts in Table 4b collected from the same runs used for Table 2?
Q2. How many instances are evaluated per task, and are the reported accuracy differences statistically significant? Could the authors also report cost distributions, uncertainty estimates, and budget-exhaustion rates? Mean costs alone may conceal substantial variation across instances.
Q3. Were the harnesses given comparable prompt-tuning efforts, interaction limits, and retry budgets? Could the authors clarify whether the observed performance differences are robust to prompt variations and whether the proposed mechanism-level explanations are supported by component ablations?
Q4. What evidence shows that the synthetic tasks measure the intended harness capabilities rather than generation or formatting artifacts? Additionally, since the comparisons with existing benchmarks use only GPT-5.6-sol, would the claimed advantages remain consistent across other model families?
3 comments
...whether the proposed mechanism-level explanations are supported by component ablations? What evidence shows that the synthetic tasks measure the intended harness capabilities rather than generation or formatting artifacts?
We provide additional behavioral analysis on 800 GPT-5.6-sol agent runs, covering four harnesses and four tasks to quantify the harness behaviors discussed in our section 4.4. In general, our tasks are capable of showing substantially different costs between context processing strategies. Some notable processing example include:
Program Execution Tracing (Appendix D.4): RLM uses two primary delegation strategies:
-
Step-based filtering (22/50 samples): The parent agent parses the query into 8 steps, filters relevant program cells for each, and assigns one step per child agent to identify candidates (one-to-many mapping). Average cost: $0.935/sample.
-
Context-based partitioning (21/50 samples): The parent agent splits the cells evenly and sends the full 8-step query to each child agent (many-to-many mapping). Average cost: $2.08/sample
Outlier Memo Detection (Appendix D.2):
-
mini-swe-agent (50/50 samples): Parses entity names to group memos by entity pairs, then uses code and templates to extract entity-relationship entries into a fact table. It programmatically detects conflicting claims and identifies outliers based on supporting fact counts. Average cost: $0.583/sample.
-
RLM: Despite having programming capabilities, RLM chunks the memos and delegates fact-table generation to child models. This repeated LLM-based parsing across chunks drives up costs to $7.23/sample.
For the more detailed analysis, please refer to our full appendix D.
would the claimed advantages to other datasets remain consistent across other model families other than GPT-5.6-sol?
Relative to existing benchmarks, LongHarness (1) separates harnesses more clearly and (2) gives a more comprehensive view of harness behavior. We run Sonnet-5.5 as a second model family, and both points hold for both models.
Table R1. Accuracy (avg. USD per instance). Each cell: GPT-5.6-sol / Sonnet-5.5. LongHarness shows the macro average over its four tasks.
| Harness | LongHarness | OOLONG-Synth | LongBench-v2 | |
|---|---|---|---|---|
| Direct | 60.0% (\0.706) | 64% (\0.394) | 62% (\0.403) | |
| mini-swe-agent | 68.5% (\0.367) | 86% (\0.106) | 68% (\0.086) | |
| OpenCode | 41.0% (\0.498) | 74% (\0.150) | 60% (\0.102) | |
| RLM | 53.5% (\0.649) | 78% (\0.231) | 62% (\0.222) | |
| ReAct | 43.0% (\0.742) | 70% (\0.500) | 68% (\0.105) |
(1) Clearer separation. Across five configurations, LongHarness accuracy spans 41.0–68.5% for GPT-5.6-sol and 25.5–79.5% for Sonnet-5.5, versus 64–86% / 50–80% on OOLONG-Synth and 60–68% / 52–62% on LongBench-v2. In addition, while on both existing benchmarks, every harness costing more than direct inference is at least as accurate, on LongHarness, three GPT-5.6-sol harnesses (OpenCode, RLM, ReAct) cost 2.3–5.5× more yet score lower.
(2) A more comprehensive view. On OOLONG-Synth, both models agree that mini-swe-agent is the best harness and direct inference the worst. LongHarness's four tasks ranks harness differently: For GPT-5.6-sol, mini-swe-agent leads on Memos, OpenCode on Tracing, and ReAct on Pairs, while direct inference ties for best on Constraint. The performance of the same harness can vary differently across tasks and models. For instance, with GPT-5.6-sol, Opencode scores 96% on Tracing but 0% on Pairs; on Memos, it scores 100% with Sonnet-5.5 versus 10% with GPT-5.6-sol.
On the scope of the motivation. Our central claim is that existing benchmarks do not adequately distinguish modern harnesses. Table 3 supports this: LongBench-v2 shows little separation, while OOLONG-Synth separates only in one direction, with the same best and worst configurations for both models. We will make the abstract’s description of this gap (“saturated accuracy”, “similar costs”) more precise.
We will add these results, including the per-task Sonnet-5.5 numbers, to Tables 2–3 and Section 4.5, and clarify the intent of this comparison in the abstract.
The authors' response has addressed most of my concerns. I have decided to raise my score to 3.
We thank the reviewer for your comments. We address your questions as follow:
Our cost calculation follow the following formula: cost=(I−C−W)rinput+Crcached+Wrwrite+Oroutput
Here, I is total input, C cached reads, W cache writes, O output including reasoning and r denotes corresponding API pricing . And sub-calls made by harnesses are also counted toward this budget. We have updated Table 4 in the appendix to show our cost breakdown. The token counts in Table 4b were collected from the exact same run used in Table 2.
For example, GPT-5.6-sol with OpenCode on Constraint Solving Search has the following mean usage per instance:
The reconstructed cost is 4(0.000750)+0.4(1.644112)+5(0.192437)+20(0.005953)=$1.7418898≈$1.74 matching Table 2. Table 4(c) additionally identifies tokens subject to higher request-length rates where applicable.
Each of our reported tasks has 50 samples totalling a 200 instance test suite. We have reported the 95% Wilson confidence interval in our updated paper (Table 4). Cost distribution and number of budget exhaustions are also added (Appendix E).
Section 4.3 uses small numerical differences to describe similar observed performance across configurations. Our central conclusions are supported by substantial accuracy differences. For GPT-5.6-sol, mini-swe-agent improves Outlier Memo Detection accuracy from 44% to 100%. This is a 56-percentage-point gain, with a paired 95% bootstrap confidence interval of [42, 70] points. On Equivalent Program Pair Search, RLM reduces accuracy from 44% to 2%. This is a 42-percentage-point drop, with a corresponding confidence interval for the drop of [28, 56] points. Both intervals exclude zero, supporting the observed directions of change.
Our efficiency findings also concern substantial differences. On Outlier Memo Detection, mini-swe-agent achieves 100% accuracy at \0.583perinstance,whileRLMachieves927.23, approximately 12.4 times the cost. These central findings do not depend on resolving small accuracy differences.
We guarantee that all harnesses receive the same vanilla prompt describing the task requirement, retry budget for each tool call, interaction limits are also set so that they can only work on the sample files without accessing gold answer or browsing the internet. For our results robustness, because of budget constraint, we additionally report the 95% Wilson confidence interval.