acceptodds
Under review as a conference paper at ICLR 2027

Harness²: Evaluating Coding-Agent Memory After a Reset

10 views

Abstract

A coding agent receives a stored code change to help repair its next program. The resulting recovery score is taken as a verdict on what the memory transfers. The harness can settle that verdict before a program reaches the verifier. On two held-out functions at a 16,384-token per-call cap, 19 of 24 episodes at extra-high reasoning effort emit no program; forced closure prevents an observed end-token failure mode on Qwen3.8-27B-FP8. We introduce Harness² (Harness of Harnesses), which starts from a failing program, excludes prior interactions and the target solution, and compares reference-derived banks with the task, seed, model, verifier, and execution limits held fixed. On 60 completed PACE-Bench task–seed units, a sibling reference change resolves 23 episodes versus 22 with no bank: 12 paired wins, 11 losses, and 37 ties, a difference of 1/60. A character-matched unrelated bank resolves 21, while direct patch application and whole-program transplant both fail on the tested cross-task transfers. Recovery is reported beside the output tokens consumed by the same evaluated episodes, excluding earlier executions that were replaced: the bank is evaluated through the harness that reads it.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
agent memory, program reconstruction, coding agents, verification, test-time computation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DZfsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Harness², a reset-based protocol for evaluating coding-agent memory under controlled execution settings. It compares reference-derived code diffs, unrelated diffs, and no memory, while examining how inference budgets and reasoning-channel termination affect recovery. The experiments primarily use PACE-Bench.

Critical strengths
  1. The paper raises a practically relevant concern about potential confounding factors introduced by agent harness configurations in memory evaluation.
  2. The paired comparisons and reporting of execution costs and failure modes provide useful experimental transparency.
Critical weaknesses
  1. Limited novelty and technical contribution. The core memory evaluation largely amounts to injecting different reference code diffs into the prompt and comparing recovery rates. The paper introduces neither a new memory mechanism nor a sufficiently differentiated evaluation methodology beyond conventional prompt ablations.

  2. The inference-budget findings are largely unsurprising. One central observation is that insufficient output-token budgets prevent reasoning models from completing their reasoning and producing code. This appears primarily to be an expected consequence of inadequate inference configuration rather than a novel scientific insight. A convincing evaluation should use appropriately tuned budgets or demonstrate performance saturation across budget settings.

  3. Limited experimental scale and representativeness. The primary memory evaluation covers only 10 PACE-Bench tasks, six seeds, and one model. The 60 task–seed units do not represent 60 independent tasks. Evaluation on more representative coding-agent benchmarks, such as SWE-bench Verified or Terminal-Bench, and additional models would considerably strengthen the work.

  4. The central claim lacks direct experimental validation. Although the paper emphasizes that harness configurations affect memory evaluation, the harness ablations and memory comparisons are largely conducted separately. A systematic harness × memory interaction study is missing, leaving the central claim about harness-dependent memory effects insufficiently demonstrated.

Questions for authors
  1. What substantive methodological contribution does Harness² provide beyond standard prompt-level ablations?

  2. Why should the output-budget limitation be considered a central research finding? Do the conclusions persist under well-tuned or performance-saturated inference budgets?

  3. Why are more representative coding benchmarks, such as SWE-bench or Terminal-Bench, not included? Can the findings generalize across models and realistic coding tasks?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer DZfs (1/3): what Harness² measures beyond a prompt ablation, and the saturated budget

Thank you for your review and for the time you spent on our paper. Here are our replies.

W1 & Q1.

"neither a new memory mechanism nor a sufficiently differentiated evaluation methodology beyond conventional prompt ablations"

Answer: Harness² adds three checks absent from a prompt ablation, and the first check shows that code memory's lead at a 16,384-token cap measures the length of the first call. First, it verifies whether an episode emits an executable program at all. At a 16,384-token per-call cap, 19 of 24 extra-high-effort episodes on two NetworkX functions end at their first call without a program, and without forced closure, 16 of 30 episodes on Qwen3.8-27B-FP8 halt at an end token inside the reasoning channel, seven of them before any program. Second, it compares memory banks within matched task–seed units under sharp finite-panel bounds that need no missing-data model. Third, it calibrates the comparison with the target task's own reference diff on the first three seeds of the same panel:

Bank against no bank, Qwen3.8-27B-FP8UnitsWinsLossesDifferenceSign-flip pp
Target's own diff (calibration), first three seeds301620.470.001
Sibling diff, first three seeds3045−0.031.00
Sibling diff, all six seeds6012110.021.00
Character-matched unrelated diff, all six seeds601011−0.021.00

On the same panel and against the same no-bank episodes, the protocol detects the change that solves the target task and finds no detectable gain from the closest cross-task diff. The revised Table 2 reports both 30-unit rows, and the contribution summary lists the three checks.

W2 & Q2.

"A convincing evaluation should use appropriately tuned budgets or demonstrate performance saturation across budget settings."

Answer: The budget finding is central because a binding cap reorders memory formats, and every conclusion of the memory panel is measured at a per-call cap that none of its calls reaches. In the cap study, at 49,152 tokens "every returned episode on the same seed groups emits one" program (lines 049–050), and every first call at this cap "closes below 49,152 tokens" (lines 316–317). The memory panel runs at the same per-call cap (lines 258–259). Across all 730 generation calls of its 60 units and three banks, no call reaches the cap, and the longest call is 39,302 tokens. The five calls of an episode share an allowance of 98,304 output tokens, identical across banks, which acts as the attempt budget of a task. It ends a similar number of unresolved episodes in every bank, 20 sibling, 18 placebo and 16 no-bank episodes of the 60 per bank, so it constrains the banks equally.

The smaller cap shows what a binding cap does to a memory comparison. Each comparison in the cap study fixes "the target, memory format, and seed" (lines 308–310). Empty memory supplies no stored record, the text note supplies an earlier episode's stored skill record, and reconstructed code supplies that episode's reconstructed program. Raising the cap removes the lead of reconstructed code:

Extra-high effort, same seedsReconstructed codeEmptyText note
16,384-token cap3/81/81/8
49,152-token cap5/85/84/8

At the smaller cap every unresolved episode has no program, so the apparent advantage of code memory reflects the length of the first call. The revised Section 4.2 reports this table alongside the PACE memory panel analysis.

AuthorsOpenReview ↗

Reply to Reviewer DZfs (2/3): tasks, models and representativeness

W3 & Q3.

"only 10 PACE-Bench tasks, six seeds, and one model", "SWE-bench Verified or Terminal-Bench"

Answer: A reset comparison needs a released failing program, a verified reference diff, and sibling tasks from the same adaptation family, all three of which PACE-Bench provides, and the revision extends the memory panel to further PACE-Bench tasks and to two further backbones. Each adaptation pair supplies a failing initial program, a mutated target environment, a passing reference program, and siblings in the same physics family. Together these define the sibling bank, the unrelated control, and the own-diff calibration. SWE-bench Verified and Terminal-Bench provide one reference solution per task. Forming sibling banks there requires grouping related issues into families that adapt one program to related changes, each with a verified fix, which the revised Limitations names as the step a new benchmark needs. The two harness settings, the per-call output cap and the reasoning-channel termination rule, exist in every harness that serves a reasoning model, and the protocol measures their effect on each backbone.

In the revision we run the three banks on all 24 further PACE-Bench tasks that qualify, every task outside the ten whose initial program fails and whose reference passes, each paired with a sibling from its family across all six physics families. At seeds 1729 and 2718 on Qwen3.8-27B-FP8, all 144 episodes return:

Bank against no bank, 24 further tasksSeedWinsLossesDifferenceSign-flip pp
Target's own diff17291210.460.003
Sibling diff1729330.001.00
Target's own diff27181500.63< 0.001
Sibling diff2718620.170.29

On these new tasks the panel again detects the target's own change, and the sibling diff shows no detectable gain at either seed or over the pooled 48 units (9 wins, 5 losses, pp = 0.42).

The inference of each panel concerns its own units, and representativeness rests on repeating the panel on new tasks, seeds and backbones. The finite-panel bounds make no independence assumption across units, and under the arm-swap invariance of its null the sign-flip reference holds "even if the original units are dependent" (Appendix A.2, lines 577–578). Beyond the submission, we also ran the reset with agent-written memory on four new seeds, adding 120 episodes:

40 units, four new seedsResolvedWins / lossesDifference
Agent-written note against raw trajectory13 against 117 / 5+0.05
Agent-written note against sibling diff13 against 143 / 4−0.025

The memory panel also runs on two further backbones on seeds 1729 and 2718, Qwen3.5-4B and Granite 4.2 30B from another model family:

Bank against no bank, 20 unitsReturnedWinsLossesSign-flip pp
Qwen3.5-4B, target's own diff201120.022
Qwen3.5-4B, sibling diff18111.00
Granite 4.2 30B, target's own diff20600.031
Granite 4.2 30B, sibling diff20101.00

No outcome of the two missing Qwen3.5-4B units changes its reading.

On Qwen3.8-27B-FP8, forced closure wins 3 units with no loss. Qwen3.5-4B also enters the closure study, never halts at an end token, and ties on all 30 pairs. Because this behavior varies across backbones, the protocol measures the closure rule per model before comparing memory banks.

AuthorsOpenReview ↗

Reply to Reviewer DZfs (3/3): the harness × memory interaction

W4.

"A systematic harness × memory interaction study is missing"

Answer: The cap study crosses the per-call cap with the memory format, and the revision adds a second crossing of the closure rule with the memory bank. Each comparison fixes the target, memory format, and seed (lines 308–310). In the second table of our reply to W2, all 19 unresolved episodes at the 16,384-token cap emit no program, so the cap acts before any memory format can, and raising it removes the lead of reconstructed code. On the PACE memory panel, seed-coupled decoding (Equation 2) locates where a smaller cap would act on each bank:

First calls cut before they close, of 60SiblingPlaceboNo bank
16,384-token cap384436
24,576-token cap14710
49,152-token cap, used in the paper000

At 16,384 tokens, the cap would cut only one arm's first call in 24 sibling–placebo, 22 sibling–no-bank and 22 placebo–no-bank pairs, against 14, 23 and 21 pairs whose outcomes differ at the cap used in the paper. A memory comparison at 16,384 tokens would therefore rest on about as many one-sided cuts as outcome differences. The revised Section 4.2 adds this table under "Budget × memory".

The second harness setting, the termination rule of the reasoning channel, is crossed with memory on the 30 closure units of Qwen3.8-27B-FP8:

30 closure units, Qwen3.8-27B-FP8Target's own diffSibling diffNo bank
Resolved with forced closure26 of 3011 of 3012 of 30
Resolved without forced closure21 of 309 of 289 of 30
Units won / lost by closure5 / 02 / 03 / 0
End-token stops without closure81615
Of these, before any program467

Without closure, the own diff wins 12 of the 30 units against no bank and loses none, and the sibling diff wins 3 and loses 2, so the calibration and the null sibling reading hold under both termination rules. The rule acts in every bank, and the own diff halves how often an end token halts an episode. One fluids unit in every column comes from a rerun after a transport fault, so the no-bank column counts one end-token stop fewer than the initial executions in our reply to W1. The revised Section 4.2 and Table 2 add this crossing.

If the measurements above address your concerns, we would be grateful if you would consider raising your score.

Thank you again for your review. We're happy to address any further questions you may have.

Reviewer_zdj6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how to evaluate memory for coding agents on repair tasks. It first argues that a recovery rate alone can mislead because the execution harness can decide failure before any program reaches the verifier, and it documents two such routes on Qwen3.8-27B-FP8 and an end token emitted inside the reasoning channel, which the authors handle by replacing it with </think> and continuing generation. With these settings fixed, the paper runs a reset protocol on 10 PACE-Bench adaptation pairs comparing three prompt conditions: the reference diff of the single sibling task in the same physics family, a character-matched diff from an unrelated family, and no diff. The three arms resolve 23, 21, and 22 episodes respectively.

Critical strengths
  • Evaluators should check whether a program was produced at all before interpreting a success rate.
Critical weaknesses
  • The title promises an evaluation of coding-agent memory, but what the paper actually tests is only loosely related to memory. The “memory bank” is a single reference diff from a sibling task, copied verbatim into the prompt. It is not written by the agent from its own experience, nothing accumulates across tasks, and there is no retrieval. Each target has exactly one donor, so a bank is one record. This setup is closer to an ablation of one-shot example prompting. The name “Harness of Harnesses” does not fit either. There is only one harness in the paper. The authors change two of its settings (output cap and the termination rule for the reasoning channel) and don't compare harnesses against each other.
  • The findings are not new. That a per-call output cap can cut off reasoning before any program is emitted is already established by the overthinking line of work. This paper re-observed on two toy functions. Forced closure, which rewrites an end token inside the reasoning channel as </think>, is in effect a workaround for a quirk of one specific model (Qwen3.8-27B-FP8). It is hard to count as a methodological contribution. The “reset” protocol is a standard controlled comparison. VibeMemBench already uses irrelevant-memory controls and held-out solvers, and the related-work section gives them one sentence without saying what this paper adds. The two propositions are, respectively, an interval obtained by setting missing binary outcomes to their extremes and the ordinary sign test. The memory comparison itself is a null result, and the sample is too small to make that null result convincing.
Questions for authors
  • What findings of this paper are unique and significant compared to existing literature?
  • In what sense does the paper evaluate coding-agent memory rather than one-shot example prompting?
  • How will the proposed method perform on substainlly more evaluation tasks?
5 comments
AuthorsOpenReview ↗

Reply to Reviewer zdj6 (1/3): memory, agent-written banks, and the name

Thank you for a detailed and technically engaged review. Your questions on what counts as memory and on the strength of the null result led to the agent-written study and the paired own-diff calibration of the revised paper. Here are the replies.

W1.1 & Q2.

The bank is one fixed reference diff, closer to one-shot example prompting than to memory.

Answer: The sibling diff is the verified solution of the closest task in the pool, the record a retrieval memory would return with perfect retrieval, and the revision extends the reset to agent-written memories distilled from prior experience. Any memory that writes, accumulates, and retrieves conditions the target episode on a record of earlier work. Within the pool, the closest such record is the verified diff from a sibling task in the same physics family. Fixing that record before generation isolates downstream memory utility from writing quality and retrieval error. We then ran agent-written memory under the same reset. For each target, one writer call distilled a sibling episode's record into a procedural note, and the note, a bounded raw trajectory of the same episode, and the sibling diff were frozen before 120 consumer episodes across four new seeds:

40 units, four new seedsResolvedWins / lossesDifference
Agent-written note against raw trajectory13 against 117 / 5+0.05
Agent-written note against sibling diff13 against 143 / 4−0.025

Neither contrast meets our decision rule, fixed before the consumer episodes ran, of a gain of at least 0.05 with one-sided p≤0.025p \le 0.025. Model-free checks show that verbatim reuse transfers nothing on these tasks, as none of 132 cross-task diffs applies cleanly as a patch and none of 132 transplanted programs passes its target verifier. The revised Section 4.3 adds this study, with its protocol in the appendix.

W1.2.

The name does not fit a paper with one harness and two of its settings.

Answer: Harness² is the outer evaluation harness that locks the agent's inner harness settings across arms, and the paper compares both harness settings and complete search harnesses. Specifically, it measures the end-token rule on three backbones, two output caps across three reasoning efforts, and four multi-step search procedures run as complete harnesses on the same tasks under a 20-call protocol, beside the five-call protocol of the memory panel (Table 2, Figure 5C). The revised abstract defines the name where it first appears.

AuthorsOpenReview ↗

Reply to Reviewer zdj6 (2/3): novelty of the harness findings and of the reset

W2.1.

Cap truncation is known from the overthinking literature and is shown here on two toy functions.

Answer: Prior overthinking work shows that extended reasoning can displace action, and Harness² shows how this truncation reorders a memory comparison. In the cap study, each paired comparison fixes the target, memory format, and seed, and raising the cap removes the lead of reconstructed code over empty memory and a text note:

Extra-high effort, same seedsReconstructed codeEmptyText note
16,384-token cap3/81/81/8
49,152-token cap5/85/84/8

Under the 16,384-token cap, the apparent advantage of code memory reflects the length of the first call. On the PACE memory panel, where no call reaches the 49,152-token cap the paper uses, seed-coupled decoding shows that a 16,384-token cap would cut only one arm's first call in 24 sibling–placebo, 22 sibling–no-bank and 22 placebo–no-bank pairs, about as many as the 14, 23 and 21 pairs whose outcomes differ at the paper's cap. The two NetworkX functions pinpoint this gate in the first generation call through matched traces.

W2.2.

Forced closure is a workaround for one model's quirk.

Answer: Every harness serving a reasoning model sets a rule for how the reasoning channel ends, and forced closure applies the closing step of documented rules to a different trigger. Budget forcing ends the reasoning channel by appending the end-of-thinking delimiter [1], and the Qwen3 thinking budget inserts the closing tag when the budget is reached [2]. Forced closure inserts the same tag when the model emits an end token inside the channel. The paper measures this rule on three backbones. Without closure, 16 of 30 episodes on Qwen3.8-27B-FP8 halt at an end token, seven of them before any program, and closure wins 3 units with no loss. Qwen3.5-4B never halts this way and all 30 of its pairs tie, and none of the 270 calls of Granite 4.2 30B ends on an end token inside the reasoning channel. On Qwen3.8-27B-FP8, the same termination appeared in 21 of 24 returned episodes across three different search procedures. Because any memory comparison inherits its harness's termination rule, the rule must be measured per backbone before banks are compared.

W2.3.

The reset is a standard controlled comparison, and VibeMemBench already has irrelevant-memory controls and held-out solvers.

Answer: The reset controls two confounds that VibeMemBench leaves open, outcome-based task selection and programless episode endings. VibeMemBench's irrelevant-memory controls and held-out solvers test transfer on the targets VibeMemBench keeps. Because it keeps a target task only when injected experience improves the task's outcome in a reference run, its direct-injection gains of 1.1 to 4.5 points on four of five held-out solvers are measured on targets selected for that gain. Harness² freezes every bank before any episode runs, retains every planned unit of the memory panel, and tallies programless endings before comparing banks, preventing selection bias and harness truncation from inflating memory scores. VibeMemBench's broader observation, that existing memory systems do not exceed memory-off in 11 of 12 pairings, agrees with ours. The revised Section 2 states this methodological distinction.

W2.4.

The propositions are a worst-case interval and the sign test.

Answer: We use the standard worst-case bound and the exact sign-flip test as reporting tools because they require no missing-data model for incomplete episodes, and the bound decided a reported result. At the analysis cutoff, two closure units had not returned, and Proposition 1 fixed the sign of the full-panel closure effect within [0.10, 0.17] without any assumption on missingness. Both units have since completed, yielding an observed difference of 0.10 inside that exact interval, with no loss. Under the arm-swap invariance of its null, Proposition 2 provides a conditional sign-flip reference that holds even when units are dependent. Because the paper's contribution is the evaluation protocol, these propositions serve as its formal reporting tools.

References

[1] Muennighoff et al. s1: Simple test-time scaling. arXiv:2501.19393, 2025.

[2] Yang et al. Qwen3 Technical Report. arXiv:2505.09388, 2025.

AuthorsOpenReview ↗

Reply to Reviewer zdj6 (3/3): resolution of the null, unique findings, and more tasks

W2.5.

The null result rests on too small a sample.

Answer: On the same panel and against the same no-bank episodes, the panel detects a large effect from the target's own diff and none from the sibling diff, and the same reading repeats on 24 further tasks (our reply to Q3) and on two further backbones.

Bank against no bank, Qwen3.8-27B-FP8UnitsWinsLossesDifferenceSign-flip pp
Target's own diff (calibration), first three seeds301620.470.001
Sibling diff, first three seeds3045−0.031.00
Sibling diff, all six seeds6012110.021.00

Across all 60 units, the 23 discordant units could resolve a difference of about 0.18, well above the observed 1/60. On Qwen3.5-4B the own diff wins 11 units and loses 2 and the sibling diff wins 1 and loses 1, and on Granite 4.2 30B the own diff wins 6 and loses none and the sibling diff wins 1 and loses none. On four new seeds, the agent-written note does not exceed the sibling diff either, 13 against 14 resolved. The revised Table 2 adds these rows.

Q1.

Which findings are unique and significant compared with existing literature?

Answer: The paper establishes three empirical findings that are new for coding-agent memory evaluation. Overthinking studies measure how extended reasoning displaces action, and the first finding adds how that truncation decides a memory comparison.

  1. Before any memory is compared, a 16,384-token per-call cap leaves 19 of 24 extra-high-effort episodes on two NetworkX functions without a program, an end token inside the reasoning channel ends 16 of 30 episodes without forced closure (seven of them before any program), and raising the cap removes the lead of one memory format.
  2. With both harness settings fixed and a per-call cap that no call reaches, the closest cross-task diff adds 1/60 on 60 units. On the panel's first three seeds, it wins 4 and loses 5 against no bank, and the target's own diff wins 16 and loses 2.
  3. Agent-written procedural notes and raw trajectories do not exceed the reference diff, and model-free verification shows that no stored diff or program works verbatim on another task.

Q3.

How does the method perform on substantially more evaluation tasks?

Answer: The revision runs the memory panel on 24 further PACE-Bench tasks, every task outside the ten that has a failing initial program and a passing reference and pairs with a sibling of its family. We run seeds 1729 and 2718 on Qwen3.8-27B-FP8, and all 144 episodes return:

Bank against no bank, 24 further tasksSeedWinsLossesDifferenceSign-flip pp
Target's own diff17291210.460.003
Sibling diff1729330.001.00
Target's own diff27181500.63< 0.001
Sibling diff2718620.170.29

The panel again detects the target's own change, and the sibling diff shows no detectable gain at either seed or over the pooled 48 units (9 wins, 5 losses, pp = 0.42). Beyond the 60 units in the submission, the agent-written study adds four new seeds and 120 episodes. Because the two harness settings belong to the generation call itself, they apply to every task suite evaluated with a reasoning model, and the protocol measures their effect per suite and backbone.

If these measurements resolve your concerns about novelty and about the null result, we would be grateful if you would consider raising your score.

Thank you again for your detailed and technically engaged review. We're happy to address any further questions you may have.

Reviewer_RBLKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on ensuring that the Harness itself is setup correctly and is not causing failures, before comparing the effect of memory. It makes the point that success rate by itself is not enough and failures that are related to the Harness need to be resolved first. With this in mind, the authors run experiments on Qwen model and show that adding a related example/memory doesn't have a significant effect on resolution rate once they fix Harness related issues.

Critical strengths
  • The paper focuses on an important problem, it is indeed not enough to consider resolution rate in agent experimentations. Analyzing the failures can point to harness related issues that should be resolved before any comparison studies. Otherwise, the harness failures may interfere with the result of an experiment.
  • Although the paper doesn't answer whether memory has an effect or not on task resolution, the experiments do prove the point that ensuring the correctness of harness is important.
Critical weaknesses
  • The paper only looks at one model and using one data set.
  • Although it makes sense that Harness matters but this is true for any experiment where all conditions need to be carefully reviewed and ensure that they don't interfere with the experiment, which obviously includes the harness. Therefore, without any other results just knowing that harness has a effect doesn't seem to be enough of a contribution.
Questions for authors
  • Have you run these experiments for other data sets? Have you ran the experiment one time or multiple times on the current data set?
  • Have you tried other models?
  • Are the harness issues identified well-known settings for working with the Qwen model or there are specific to this harness?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer RBLK (1/2): models, datasets and repeated runs

Thank you for an insightful review. Your framing of harness failures as a check that precedes any comparison became a paragraph of Section 4.2 in the revised paper. Here are the replies.

W1, Q1.1 & Q2.

The paper looks at one model and one dataset.

Answer: The harness findings span two models and two task sources, and the revision runs the memory comparison on two further models, one from another model family, and on further PACE-Bench tasks. A memory comparison after a reset needs a released failing program, a verified reference diff and sibling adaptations of the same program, which PACE-Bench provides and issue-level suites such as SWE-bench Verified provide only after related issues are grouped into families. Qwen3.8-27B-FP8 and Qwen3.5-4B both enter the closure study on the same 30 PACE-Bench units. On the 27B model, 16 of 30 episodes without closure halt at an end token inside the reasoning channel, and closure wins 3 units with no loss. The 4B model never halts this way, and all 30 of its pairs tie. The cap study runs on two NetworkX functions, and the memory comparison runs on PACE-Bench adaptation pairs with Qwen3.8-27B-FP8. Beyond the submission, we ran the memory panel on seeds 1729 and 2718 of the ten closure pairs on Qwen3.5-4B, against that model's stored no-bank episodes, and on Granite 4.2 30B with all three banks:

Bank against no bank, 20 unitsReturnedWinsLossesBound over 20 unitsSign-flip pp
Qwen3.5-4B, target's own diff20112[0.45, 0.45]0.022
Qwen3.5-4B, sibling diff1811[0.00, 0.10]1.00
Granite 4.2 30B, target's own diff2060[0.30, 0.30]0.031
Granite 4.2 30B, sibling diff2010[0.05, 0.05]1.00

Both panels detect the target's own change and find no detectable gain from the sibling diff, and no outcome of the two missing Qwen3.5-4B units changes its reading.

On the 24 further PACE-Bench tasks that qualify for a reset comparison, Qwen3.8-27B-FP8 gives the same reading over 48 units at seeds 1729 and 2718, where the own diff wins 27 and loses 1 against no bank, and the sibling diff wins 9 and loses 5 (sign-flip pp = 0.42).

We also ran the reset with agent-written memory across four new seeds:

40 units, four new seedsResolvedWins / lossesDifference
Agent-written note against raw trajectory13 against 117 / 5+0.05
Agent-written note against sibling diff13 against 143 / 4−0.025

The revised Section 4.3 adds these studies.

Q1.2.

Was the experiment run once or several times?

Answer: The three banks of the memory panel run across six seeds per task, and the agent-written study adds four new seeds. The 60 units comprise ten tasks across six seeds, paired across all three banks, with outcomes reported by physics family and by seed (Table 5 of the revised paper).

AuthorsOpenReview ↗

Reply to Reviewer RBLK (2/2): what the paper measures beyond the harness effect, and whether the issues are Qwen-specific

W2.

Harness correctness matters in any experiment, so knowing that the harness has an effect is not enough.

Answer: Beyond showing that the harness matters, the paper locates the failure in the first generation call, measures how often it removes the program, and shows that it reorders a memory comparison. At a 16,384-token cap, 19 of 24 extra-high-effort episodes on two NetworkX functions end their first call with no program, and without forced closure 16 of 30 episodes on Qwen3.8-27B-FP8 halt at an end token inside the reasoning channel. On the same seeds, raising the cap removes the lead of reconstructed code over empty memory and a text note:

Extra-high effort, same seedsReconstructed codeEmptyText note
16,384-token cap3/81/81/8
49,152-token cap5/85/84/8

With the harness fixed, on the first three seeds of the memory panel and against the same no-bank episodes, the target's own reference diff wins 16 units and loses 2, and the sibling diff wins 4 and loses 5. The revised Section 4.2 turns this into a four-step check before any memory comparison: count calls that emit no program, raise the per-call cap until no call reaches it, measure the reasoning-channel rule on each backbone, and calibrate the panel with the target's own diff.

Q3.

Are the harness issues known settings for Qwen models or specific to this harness?

Answer: Both issues arise under the model's documented settings, which our harness applies unchanged, and the end-token stop depends on the backbone. The Qwen3.8 model card recommends allocating up to 262,144 reasoning tokens for agentic tasks, so a 16,384-token per-call cap binds whenever reasoning runs longer, and the 49,152-token cap of the memory panel is one that none of its calls reaches. Each call applies the model's official chat template and thinking-mode sampling (temperature 1.0, top-p 0.95) and stops on the model's own end-token IDs through the standard Transformers generation call. Premature end-token stops occur on Qwen3.8-27B-FP8 across three different search procedures (21 of 24 returned episodes) and never on Qwen3.5-4B or Granite 4.2 30B. Rules that close the reasoning channel are documented practice, including budget forcing [1] and the Qwen3 thinking budget [2], and every memory evaluation inherits whichever termination rule its harness applies.

We hope the further backbones and the four-step check answer your questions, and we would be grateful if you would consider raising your score.

Thank you again for your insightful review. We're happy to address any further questions you may have.

References

[1] Muennighoff et al. s1: Simple test-time scaling. arXiv:2501.19393, 2025.

[2] Yang et al. Qwen3 Technical Report. arXiv:2505.09388, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.