- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Harness², a reset-based protocol for evaluating coding-agent memory under controlled execution settings. It compares reference-derived code diffs, unrelated diffs, and no memory, while examining how inference budgets and reasoning-channel termination affect recovery. The experiments primarily use PACE-Bench.
Critical strengths
- The paper raises a practically relevant concern about potential confounding factors introduced by agent harness configurations in memory evaluation.
- The paired comparisons and reporting of execution costs and failure modes provide useful experimental transparency.
Critical weaknesses
-
Limited novelty and technical contribution. The core memory evaluation largely amounts to injecting different reference code diffs into the prompt and comparing recovery rates. The paper introduces neither a new memory mechanism nor a sufficiently differentiated evaluation methodology beyond conventional prompt ablations.
-
The inference-budget findings are largely unsurprising. One central observation is that insufficient output-token budgets prevent reasoning models from completing their reasoning and producing code. This appears primarily to be an expected consequence of inadequate inference configuration rather than a novel scientific insight. A convincing evaluation should use appropriately tuned budgets or demonstrate performance saturation across budget settings.
-
Limited experimental scale and representativeness. The primary memory evaluation covers only 10 PACE-Bench tasks, six seeds, and one model. The 60 task–seed units do not represent 60 independent tasks. Evaluation on more representative coding-agent benchmarks, such as SWE-bench Verified or Terminal-Bench, and additional models would considerably strengthen the work.
-
The central claim lacks direct experimental validation. Although the paper emphasizes that harness configurations affect memory evaluation, the harness ablations and memory comparisons are largely conducted separately. A systematic harness × memory interaction study is missing, leaving the central claim about harness-dependent memory effects insufficiently demonstrated.
Questions for authors
-
What substantive methodological contribution does Harness² provide beyond standard prompt-level ablations?
-
Why should the output-budget limitation be considered a central research finding? Do the conclusions persist under well-tuned or performance-saturated inference budgets?
-
Why are more representative coding benchmarks, such as SWE-bench or Terminal-Bench, not included? Can the findings generalize across models and realistic coding tasks?
5 comments
Reply to Reviewer DZfs (2/3): tasks, models and representativeness
W3 & Q3.
"only 10 PACE-Bench tasks, six seeds, and one model", "SWE-bench Verified or Terminal-Bench"
Answer: A reset comparison needs a released failing program, a verified reference diff, and sibling tasks from the same adaptation family, all three of which PACE-Bench provides, and the revision extends the memory panel to further PACE-Bench tasks and to two further backbones. Each adaptation pair supplies a failing initial program, a mutated target environment, a passing reference program, and siblings in the same physics family. Together these define the sibling bank, the unrelated control, and the own-diff calibration. SWE-bench Verified and Terminal-Bench provide one reference solution per task. Forming sibling banks there requires grouping related issues into families that adapt one program to related changes, each with a verified fix, which the revised Limitations names as the step a new benchmark needs. The two harness settings, the per-call output cap and the reasoning-channel termination rule, exist in every harness that serves a reasoning model, and the protocol measures their effect on each backbone.
In the revision we run the three banks on all 24 further PACE-Bench tasks that qualify, every task outside the ten whose initial program fails and whose reference passes, each paired with a sibling from its family across all six physics families. At seeds 1729 and 2718 on Qwen3.8-27B-FP8, all 144 episodes return:
| Bank against no bank, 24 further tasks | Seed | Wins | Losses | Difference | Sign-flip |
|---|---|---|---|---|---|
| Target's own diff | 1729 | 12 | 1 | 0.46 | 0.003 |
| Sibling diff | 1729 | 3 | 3 | 0.00 | 1.00 |
| Target's own diff | 2718 | 15 | 0 | 0.63 | < 0.001 |
| Sibling diff | 2718 | 6 | 2 | 0.17 | 0.29 |
On these new tasks the panel again detects the target's own change, and the sibling diff shows no detectable gain at either seed or over the pooled 48 units (9 wins, 5 losses, = 0.42).
The inference of each panel concerns its own units, and representativeness rests on repeating the panel on new tasks, seeds and backbones. The finite-panel bounds make no independence assumption across units, and under the arm-swap invariance of its null the sign-flip reference holds "even if the original units are dependent" (Appendix A.2, lines 577–578). Beyond the submission, we also ran the reset with agent-written memory on four new seeds, adding 120 episodes:
| 40 units, four new seeds | Resolved | Wins / losses | Difference |
|---|---|---|---|
| Agent-written note against raw trajectory | 13 against 11 | 7 / 5 | +0.05 |
| Agent-written note against sibling diff | 13 against 14 | 3 / 4 | −0.025 |
The memory panel also runs on two further backbones on seeds 1729 and 2718, Qwen3.5-4B and Granite 4.2 30B from another model family:
| Bank against no bank, 20 units | Returned | Wins | Losses | Sign-flip |
|---|---|---|---|---|
| Qwen3.5-4B, target's own diff | 20 | 11 | 2 | 0.022 |
| Qwen3.5-4B, sibling diff | 18 | 1 | 1 | 1.00 |
| Granite 4.2 30B, target's own diff | 20 | 6 | 0 | 0.031 |
| Granite 4.2 30B, sibling diff | 20 | 1 | 0 | 1.00 |
No outcome of the two missing Qwen3.5-4B units changes its reading.
On Qwen3.8-27B-FP8, forced closure wins 3 units with no loss. Qwen3.5-4B also enters the closure study, never halts at an end token, and ties on all 30 pairs. Because this behavior varies across backbones, the protocol measures the closure rule per model before comparing memory banks.
Reply to Reviewer DZfs (3/3): the harness × memory interaction
W4.
"A systematic harness × memory interaction study is missing"
Answer: The cap study crosses the per-call cap with the memory format, and the revision adds a second crossing of the closure rule with the memory bank. Each comparison fixes the target, memory format, and seed (lines 308–310). In the second table of our reply to W2, all 19 unresolved episodes at the 16,384-token cap emit no program, so the cap acts before any memory format can, and raising it removes the lead of reconstructed code. On the PACE memory panel, seed-coupled decoding (Equation 2) locates where a smaller cap would act on each bank:
| First calls cut before they close, of 60 | Sibling | Placebo | No bank |
|---|---|---|---|
| 16,384-token cap | 38 | 44 | 36 |
| 24,576-token cap | 14 | 7 | 10 |
| 49,152-token cap, used in the paper | 0 | 0 | 0 |
At 16,384 tokens, the cap would cut only one arm's first call in 24 sibling–placebo, 22 sibling–no-bank and 22 placebo–no-bank pairs, against 14, 23 and 21 pairs whose outcomes differ at the cap used in the paper. A memory comparison at 16,384 tokens would therefore rest on about as many one-sided cuts as outcome differences. The revised Section 4.2 adds this table under "Budget × memory".
The second harness setting, the termination rule of the reasoning channel, is crossed with memory on the 30 closure units of Qwen3.8-27B-FP8:
| 30 closure units, Qwen3.8-27B-FP8 | Target's own diff | Sibling diff | No bank |
|---|---|---|---|
| Resolved with forced closure | 26 of 30 | 11 of 30 | 12 of 30 |
| Resolved without forced closure | 21 of 30 | 9 of 28 | 9 of 30 |
| Units won / lost by closure | 5 / 0 | 2 / 0 | 3 / 0 |
| End-token stops without closure | 8 | 16 | 15 |
| Of these, before any program | 4 | 6 | 7 |
Without closure, the own diff wins 12 of the 30 units against no bank and loses none, and the sibling diff wins 3 and loses 2, so the calibration and the null sibling reading hold under both termination rules. The rule acts in every bank, and the own diff halves how often an end token halts an episode. One fluids unit in every column comes from a rerun after a transport fault, so the no-bank column counts one end-token stop fewer than the initial executions in our reply to W1. The revised Section 4.2 and Table 2 add this crossing.
If the measurements above address your concerns, we would be grateful if you would consider raising your score.
Thank you again for your review. We're happy to address any further questions you may have.
Reply to Reviewer DZfs (1/3): what Harness² measures beyond a prompt ablation, and the saturated budget
Thank you for your review and for the time you spent on our paper. Here are our replies.
W1 & Q1.
Answer: Harness² adds three checks absent from a prompt ablation, and the first check shows that code memory's lead at a 16,384-token cap measures the length of the first call. First, it verifies whether an episode emits an executable program at all. At a 16,384-token per-call cap, 19 of 24 extra-high-effort episodes on two NetworkX functions end at their first call without a program, and without forced closure, 16 of 30 episodes on Qwen3.8-27B-FP8 halt at an end token inside the reasoning channel, seven of them before any program. Second, it compares memory banks within matched task–seed units under sharp finite-panel bounds that need no missing-data model. Third, it calibrates the comparison with the target task's own reference diff on the first three seeds of the same panel:
On the same panel and against the same no-bank episodes, the protocol detects the change that solves the target task and finds no detectable gain from the closest cross-task diff. The revised Table 2 reports both 30-unit rows, and the contribution summary lists the three checks.
W2 & Q2.
Answer: The budget finding is central because a binding cap reorders memory formats, and every conclusion of the memory panel is measured at a per-call cap that none of its calls reaches. In the cap study, at 49,152 tokens "every returned episode on the same seed groups emits one" program (lines 049–050), and every first call at this cap "closes below 49,152 tokens" (lines 316–317). The memory panel runs at the same per-call cap (lines 258–259). Across all 730 generation calls of its 60 units and three banks, no call reaches the cap, and the longest call is 39,302 tokens. The five calls of an episode share an allowance of 98,304 output tokens, identical across banks, which acts as the attempt budget of a task. It ends a similar number of unresolved episodes in every bank, 20 sibling, 18 placebo and 16 no-bank episodes of the 60 per bank, so it constrains the banks equally.
The smaller cap shows what a binding cap does to a memory comparison. Each comparison in the cap study fixes "the target, memory format, and seed" (lines 308–310). Empty memory supplies no stored record, the text note supplies an earlier episode's stored skill record, and reconstructed code supplies that episode's reconstructed program. Raising the cap removes the lead of reconstructed code:
At the smaller cap every unresolved episode has no program, so the apparent advantage of code memory reflects the length of the first call. The revised Section 4.2 reports this table alongside the PACE memory panel analysis.