acceptodds
Under review as a conference paper at ICLR 2027

ProContract: Decoupling Responsibility and Progress from the Harness in Long-Horizon Agents

Abstract

Long-horizon agents are increasingly used to turn user-assigned responsibility into artifacts over hours of execution. Yet even with capable underlying models, their reliability is undermined by two persistent failure modes: responsibility dilution and progress misjudgment. Both arise because long-running agents continuously rewrite their execution state, such as delegating subtasks, compacting context, restarting, and even modifying their harness, while the responsibility and its progress remain entangled with that state. We introduce PROCONTRACT, whose progress-contract kernel independently keeps responsibilities in a Contract outside the agent and tracks progress against that Contract through authorized judgments. The agent’s harness is deliberately disentangled from the kernel, so changes in execution state cannot alter assigned responsibilities or interfere with progress judgment. This separation also makes self-improvement an ordinary responsibility: the agent can rewrite its own harness while the kernel preserves the task it was asked to carry out. With GPT-5.6 Terra, PROCONTRACT raises the mean pass rate from 72.35% to 77.24% over the same model’s public mini-SWE-agent run. The gain comes from the bottom: the number of tasks below 10% halves. A version written by the agent itself produced a successor that ran all 200 tasks under the same kernel. With GPT-5.6 Sol, PROCONTRACT raises the mean from 69.89% to 75.25% over the model’s entry on the official ProgramBench leaderboard. With GPT-5.2 on Terminal-Bench 2.0, released after that model’s knowledge cutoff, it passes 65.7% of tasks, above all four public harnesses evaluated with GPT-5.2.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM agents, agent harnesses, self improvement

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_Gpd9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Long-horizon agents turn user-assigned responsibility into artifacts over hours of execution, but they often lose track of the responsibility because it and its progress are entangled with the execution state the agent continuously rewrites, producing two failure modes the paper calls responsibility dilution and progress misjudgment. ProContract introduces a progress-contract kernel that keeps responsibilities in a Contract outside the agent and records progress only through authorized judgments, so changes in execution state cannot alter assigned responsibilities or interfere with progress judgment. The agent's harness is deliberately disentangled from the kernel, which also makes self-improvement an ordinary responsibility: the agent can rewrite its own harness while the kernel preserves the task it was asked to carry out.

Critical strengths
  1. The paper names two long-horizon failure modes and decouples responsibility and progress from the harness to address them. It identifies responsibility dilution and progress misjudgment as failures that arise because the responsibility and its progress are entangled with the execution state the agent continuously rewrites. ProContract keeps each responsibility in a Contract that only the kernel can change, has the harness read the Contract but never record progress itself, and admits progress only through a verifier's authorized judgment, so changes in execution state cannot alter assigned responsibilities or interfere with progress judgment.
  2. Progress is recorded only through a careful per-acceptance verifier rule. Progress enters only as a verifier's judgment on a frozen artifact, recorded by the kernel as an acceptance, and the kernel checks the judgment against the state in which it takes effect, so a diagnostic check run by the harness is not progress, a judgment made under stale requirements is rejected, and a judgment of one artifact never transfers to another.
Critical weaknesses
  1. Every baseline, including the official leaderboard entries and the Terra comparison, runs mini-SWE-agent, a minimal harness, while ProContract is a full system built on OpenCode with structured tool results, observation packing, diagnostic compaction, and a controller, and it spends 2.5 times as much per task. The comparison therefore varies the harness implementation, the tool surface, the context-management strategy, and the cost at the same time as the kernel, so it is not an exact comparison that isolates the kernel.
  2. The headline comparison depends on which reasoning-effort setting the baseline uses. The main gain is reported against GPT-5.6 Sol at xhigh effort, which scores 69.89, but Table 6, which the paper calls the complete third-party comparison population, reports the same model at max effort running plain mini-SWE-agent at 77.64, above ProContract's 75.25. The claim that ProContract Sol has the highest mean pass rate holds only for the Table 5 population, and the main text uses the lower-scoring baseline setting without addressing the higher one.
Questions for authors
  1. Mini-SWE-agent differs from ProContract in the harness implementation, the tool surface, the context-management strategy, and the cost, not only in the kernel. Please run ProContract's own OpenCode harness with the kernel disabled, keeping the same runtime, tools, observation packing, diagnostic compaction, and cost envelope, but letting the harness mark its own tasks done, and report the 200-task distribution. This is the only comparison that isolates the kernel from the harness implementation, and the framework already supports it.
  2. Please add GPT-5.6 Sol at xhigh effort to Table 6. The headline gain is reported against Sol at xhigh (69.89), but Table 6 reports Sol at max effort running plain mini-SWE-agent at 77.64, above ProContract's 75.25, and the two tables use different effort settings for the same model. Please report Sol at xhigh within the Table 6 population as well, so the baseline is matched in reasoning effort, and state whether the claim that ProContract Sol has the highest mean pass rate still holds under the complete third-party population.
  3. Please report whether the comparison is fair given that ProContract receives per-subtask verifier feedback. In the default setting a model solves the task on its own, whereas ProContract submits artifacts to a verifier and receives accept or withdraw judgments during execution, which mini-SWE-agent does not. A higher pass rate under this additional feedback would be expected regardless of the kernel, so please run a feedback-matched baseline that gives the baseline harness the same submit-and-judge interactions and the same accept or withdraw signals but without the kernel's responsibility-progress decoupling, and report how much of the gain remains.
8 comments
AuthorsOpenReview ↗

Response to Reviewer Gpd9 (1/2)

Dear reviewer Gpd9,

Thank you for recognizing that the paper names two long-horizon failure modes and decouples responsibility and progress from the harness to address them, and that progress is recorded only through a careful per-acceptance rule. Your three requests are precise: the first two are answered here, the third in a second comment. Each answer stands on its own; the passages behind it are set in blue in the revised paper.

1. A control that isolates ProContract from the harness

Q1 (summary): Every baseline runs mini-SWE-agent, while ProContract is a full system that also spends more. Run ProContract's own harness with the kernel disabled, on the same runtime and tools, letting the harness mark its own tasks done, and report the 200-task distribution. TL;DR: On one executable, with the same model, the same six hours and no limit on requests in either arm, the agent that marks its own task done scores 71.22% and ProContract scores 76.28%: +5.07 points (95% interval 3.70 to 6.54), higher on 154 of the 200 tasks and lower on 40. The submission's comparison with the public run of the same model gave 5.33.


Why the submission compared with public runs. A public leaderboard run is produced by a third party, so we cannot tune it, and it holds the model fixed. Your control also holds the runtime and the tools fixed.

What we ran. We now build ProContract on Codex. "Off" is the same binary with ProContract disabled and Codex's own mode for long tasks, /goal, enabled: the agent keeps a persistent goal, continues automatically, and marks its task done itself.

200 tasks, GPT-6 Luna, six hoursProContract off (Codex /goal)ProContract onChange
Mean pass rate71.22%76.28%+5.07 [3.70, 6.54]
Median task76.9%82.0%+5.1
10th percentile of tasks+10.2
90th percentile of tasks+1.6
Tasks below 50%3724−13
Tasks at or above 95%2232+10
Tasks resolved03+3
Model requests per task1561,413
Hours used0.85.9

The cost envelope. Both arms have the same allowance: six hours and unlimited requests. The baseline spends less because it declares its task done after a median of 41 minutes. That difference is the effect we measure, not a difference in budget. Spending more does not by itself raise the score: across the 199 scored runs of ProContract with GPT-6.1 Sol, the time used is uncorrelated with the pass rate (−0.04). And where a task can be finished, ProContract spends less: on Terminal-Bench 2.0 an attempt that ends with the verifier's acceptance takes a median of 48 requests and costs $1.01 on average, against 91.5 requests and $2.43 for an attempt that works to the limit.

Between off and on. A harness that marks its own tasks done is /goal. The systems in between, with the worker's "done" merely refused, or with the verifier kept and the worker free to end, are in our second comment.

In the revised paper: Table 1; Section 4.1, lines 360–364; Section 4.3, lines 396–398 and 422–423; Appendix C.9.

2. Reasoning effort, and the highest-mean claim

Q2 (summary): The headline gain is against GPT-5.6 Sol at xhigh (69.89), but the same model at max effort scores 77.64 with mini-SWE-agent, above ProContract's 75.25. Does the highest-mean claim hold on the complete third-party population? TL;DR: The submission's comparison was matched in effort: ProContract and the public run both used xhigh (75.25% against 69.89%); the 77.64% entry is a different setting, max. In the revision every run of ours is at max effort, and against the complete public leaderboard of 66 entries ProContract with GPT-6.1 Sol has the highest mean pass rate: 89.70% against 87.79% for the best entry, at 62% of that entry's cost per task.


ProgramBench, 200 tasksMean pass rateTasks at or above 95%Resolved
GPT-6.1 Sol, public run at max effort85.25%926
GPT-6 Astra, the best public entry of the same model family85.42%10011
ProContract with GPT-6.1 Sol, max effort89.70%11014

The cost of ProContract counts every model request of every role, the verifier included.

In the revised paper: Section 4.2, lines 371–376, and Figure 5; the complete leaderboard is Table 17 in Appendix C.9.

AuthorsOpenReview ↗

Response to Reviewer Gpd9 (2/2)

3. Verifier feedback

Q3 (summary): ProContract receives verifier feedback during execution that the baseline does not. With a feedback-matched baseline, how much of the gain remains?

TL;DR: All of it remains with no feedback at all: when the worker's "done" is answered by one fixed sentence, with no verifier and no information about its work, the score is higher than plain /goal on 26 of 30 tasks (median +5.4 points). The gain therefore comes from the task staying open, not from the verifier's feedback. With the verifier's assessments given in both arms, the arm in which the worker may still end the task scores 5.5 points above /goal, and the arm in which it may not scores 7.1 above; the 1.6 points between them (95% interval −2.4 to 6.6, 25 tasks) are what taking the end away from the worker adds.


Why the verifier is part of the system and not an advantage in information. The verifier sees no hidden test. It runs the reference program, which the worker also has, on the worker's frozen candidate. The baseline makes the same kind of observation itself: 183 of its 197 completion messages cite its own comparison with the reference.

The four systems (GPT-6 Luna, one panel of 40 tasks):

SystemFeedback to the workerWho may end the taskResult
Codex /goalnonethe workerstops after a median of 0.5 hours
/goal, every "done" refusedone fixed sentence: "Continue working on the same goal"no onehigher than plain /goal on 26 of 30 tasks, median +5.4 points
ProContract, the worker free to endthe verifier's assessmentthe worker, after an assessment5.5 points above /goal (2.3 to 9.1); the worker ends 18 of 25 attempts
ProContractthe verifier's assessmentonly the verifier7.1 points above /goal (2.2 to 12.9); 1.6 above the row before (−2.4 to 6.6)

The third row already runs under the kernel. There a turn's end does not close the task, as it does in /goal: the kernel freezes the candidate and the verifier assesses it. That is why the row keeps the gain. What it lacks is the last step: the worker ends 18 of the 25 attempts, at a median of 4.4 hours, and none of the 18 programs passes every hidden test.

What the kernel and the verifier add is integrity: the task stays the user's task. Under the loop the worker rewrites its goal in 5 of 32 runs, which is what we call responsibility dilution, and no run can close. Under ProContract no task is rewritten in 200 runs. On Terminal-Bench 2.0, 89 of 267 attempts close on the verifier's acceptance, and 84% of them pass, against 57% of those that run to the time limit.

Why the judgment cannot be the worker's. Shown a counterexample, the worker takes back its own "finish" 47 times out of 49. With none in hand, it still ends after 16 of the 49 assessments that say the candidate has not been accepted.

In the revised paper: Section 4.3, lines 429–445, and Figure 6d; Section 2, lines 163–170; Appendix C.11.

AuthorsOpenReview ↗

Follow-up to Reviewer Gpd9

Dear Reviewer Gpd9,

Thank you again for your review. For your three questions we ran these experiments; all are in the revised paper.

  • A control that isolates ProContract (Q1). ProContract switched off on the same executable, 200 tasks: +5.07 points [3.70, 6.54]; higher on 154 tasks, lower on 40 (Table 1).
  • Effort and the highest-mean claim (Q2). Every run is at max effort: 89.70% mean pass rate against 87.79% for the best of 66 public entries (Section 4.2).
  • Verifier feedback (Q3). Four cells on one panel of 40 tasks: keeping the task open carries the gain with no feedback at all, higher on 26 of 30 tasks (Section 4.3).
  • Decoupling from the harness, which you noted. We replaced the harness under a running task: the Contract, its history and its allowance are identical before and after (Section 4.4; Appendix C.6).

Do these answer your questions? If anything remains open, or there is a further control you would like to see, please tell us; we will answer promptly.

With thanks,

The authors

Reviewer_ZRg7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the problem of long-term task/subtask management in long-horizon agents. They identify and define the notion of "responsibilities" as smaller, verifiable subtasks that a user can issue, and progress management as long-term tracking of these subtasks without losing context. The authors claim that better memory layers alone can't address this and propose a deterministic orchestration layer called a kernel to offload this task. They show that the kernel can address the responsibility dilution/tracking problem, help guide harness improvement, and also benefit stronger frontier models on benchmarks created after the model cutoff date.

Critical strengths
  1. Problem formulation. The formalization of responsibilities is interesting and could help guide and provide a framework for future work on subtask management for long-running tasks. The paper also has claudeisms and weird writing in places.
  2. Deterministic program as the kernel. Having a deterministic program as the kernel to keep track of responsibilities makes for a more efficient system.
  3. Helps self-improvement and frontier models. The proposed kernel can facilitate self-improvement and also get some performance benefit for stronger frontier model backbones.
Critical weaknesses
  1. Bad presentation. There are no tables in the main text, and the main result is a plot that is a weird histogram that does support part of the claims (helping the worst-performing task tail), but it was hard to interpret and doesn't have concrete numbers. Having a main results table would just objectively be better.
  2. Unfair baselines. mini-SWE-agent is too simplistic a harness for long-horizon agent tasks. The authors claim that persistent memory doesn't solve the problems of responsibility dilution (which I interpret as just the agent losing track of subtasks), but this should be validated by showing a baseline with memory not performing as well as ProContract. Also, the citation used (https://docs.temporal.io/workflow-execution) doesn't even cover the claim the authors are making.
  3. Missing baselines. Another obvious missing baseline is an RLM-like [1] agent or a general orchestrator [2] worker setup where the job of the root/orchestrator is centralized control and task accounting, while subcall LLMs or workers do individual sub-tasks. This turns the long-horizon task into

[1] Zhang, Alex L., Tim Kraska, and Omar Khattab. "Recursive language models." arXiv preprint arXiv:2512.24601 (2025). [2] Samanta, Ankur, et al. "Learning What to Investigate Next: Meta-Reasoning for Long-Horizon Research Agents." arXiv preprint arXiv:2610.02525 (2026).

Questions for authors
  1. Why is a histogram of pass rates your main evaluation plot, and why are there no tables comparing your approach to other baselines?
  2. Why don't you have stronger memory-based or hierarchical harnesses (basically an LLM acting as the kernel, e.g., RLM or orchestrator-worker setups) as baselines to compare against ProContract?
6 comments
AuthorsOpenReview ↗

Response to Reviewer ZRg7 (1/2)

Dear Reviewer ZRg7,

Thank you for recognizing three strengths: the formalization of responsibilities, which you note could provide a framework for future work on long-running tasks; a deterministic program as the kernel; and that the kernel facilitates self-improvement and benefits stronger frontier models. The revision has the table and the two baselines you ask for. Each answer below stands on its own; the passages behind it are set in blue in the revised paper.

1. A table as the main result

Q1 (summary): Why is a histogram the main result, and why are there no tables comparing with baselines? TL;DR: The main result is now a table of five paired comparisons, each against a baseline that runs the same model, and ProContract is higher in every one, by 4.45 to 7.91 points. The histogram was there to show where the gain sits, at the bottom of the distribution; the table keeps that in its "below 50%" column.


ModelBaselinePaired tasksBaseline mean (%)ProContract mean (%)Gain [95% interval]Higher / lowerTasks below 50%Tasks at or above 95%Resolved
GPT-6 LunaCodex /goal, same executable20071.2276.28+5.07 [3.70, 6.54]154 / 4037 → 2422 → 320 → 3
GPT-6 LunaCodex /goal, stock release20071.2076.28+5.09 [3.78, 6.45]158 / 3736 → 2416 → 321 → 3
GPT-6 LunaRecursive Language Models20068.5676.28+7.73 [5.84, 9.75]168 / 2841 → 2419 → 321 → 3
GPT-6.1 Solmini-SWE-agent, public entry20085.2589.70+4.45 [2.37, 6.75]131 / 5519 → 992 → 1106 → 14
GPT-5.6 Lunamini-SWE-agent, public entry20068.2976.20+7.91 [5.46, 10.59]158 / 4241 → 2718 → 360 → 3

A score is the fraction of hidden tests that the final program passes; a task is resolved when it passes all of them. Arrows give baseline → ProContract. The first three rows are controlled: same model, six hours, unlimited requests.

In the revised paper: Table 1 and Figure 6; every system on every task in Appendices C.9 and C.10.

2. A stronger baseline, and memory

Q2a (summary): mini-SWE-agent is too simple a baseline; the claim that persistent memory does not solve the problem should be validated against a baseline with memory, and the citation given does not support it. TL;DR: We added Codex in /goal mode, a production agent that keeps a persistent goal and continues automatically. It stops in the same way: it declares 197 of 200 tasks complete after a median of 36 minutes, and one program passes every hidden test. ProContract is 5.07 points higher on the same executable. The early stop is not a failure of memory: these agents stop with the whole conversation still in context.


Why mini-SWE-agent. It is the benchmark's reference harness: all 66 public entries run it, the best reaches 87.79%, and its runs are produced by a third party. It stays in two rows of the table; the first three rows use stronger baselines.

Harness (six hours allowed)ModelMedian time before it stopsHow it stops
mini-SWE-agent, public runGPT-6 Luna24 minutessubmits on its own
mini-SWE-agent, public runGPT-6.1 Sol65 minutessubmits on its own
Codex /goalGPT-6 Luna36 minutesmarks its goal complete
Recursive Language ModelsGPT-6 Luna90 minutesthe root model returns
ProContractGPT-6 Luna6 hoursthe time limit; the task stays open

Why memory is not the cause. 582 of 583 public GPT runs stop on their own with the whole conversation still in the model's context: nothing had been forgotten. In 62 of the 197 /goal completion messages the agent names the missing behavior and closes the task in the same message, which is what we call responsibility dilution. And when its "done" is refused and nothing else changes, the same agent finds and fixes failures for a median of 5.3 more hours. The agent remembers the task; it is the one who decides that it is done.

The citation. We cited the workflow documentation for what durable workflows do, which is to keep execution alive across failures. The claim about memory is ours, and it rests on the measurements above.

In the revised paper: Section 2, lines 134–178; Section 4.3, lines 429–433; Section 5, lines 505–509.

AuthorsOpenReview ↗

Response to Reviewer ZRg7 (2/2)

3. Hierarchical baselines

Q2b (summary): Compare with an RLM-like or orchestrator-worker system, in which a root LLM acts as the kernel. TL;DR: We ran Recursive Language Models on all 200 tasks: 68.56%, against 71.22% for Codex /goal and 76.28% for ProContract. ProContract is 7.73 points higher (95% interval 5.84 to 9.75). A hierarchy does not change who says "done": the root model returns by its own choice after a median of 90 minutes of the six hours.


System (GPT-6 Luna, 200 tasks, six hours)Mean pass rateGain of ProContract [95% interval]Tasks higher / lowerMedian time before it stops
Recursive Language Models68.56%+7.73 [5.84, 9.75]168 / 2890 minutes
Codex /goal, same executable71.22%+5.07 [3.70, 6.54]154 / 4041 minutes
ProContract76.28%6 hours

What we ran. We adapted the authors' implementation [1], keeping its Python environment, its recursive sub-calls and its return mechanism, and ran it with GPT-6 Luna at max effort and six hours per task. In 26 runs its controller exited on an error from the model interface, and the program it left is scored (52.3% on average). The 174 runs that returned normally average 71.0%, and ProContract is 6.3 points higher on those tasks.

Why a root model is not a kernel. A root model that tracks the task is still the system that certifies its own work. It is also a good execution strategy, higher than ProContract on 28 of the 200 tasks, and such a strategy belongs in the harness, which ProContract leaves free to change. The kernel is the part that should not be a model. The orchestrator of [2] chooses what to investigate next, which is also a harness decision; the related work cites both.

In the revised paper: Table 1 and Figure 6d; Section 5, lines 505–509; Appendix C.5.

References:

  1. Zhang, Kraska and Khattab. "Recursive Language Models." 2025. Arxiv 2512.24601
  2. Samanta, Efroni, Sajda, Hassani and Goyal. "Learning What to Investigate Next: Meta-Reasoning for Long-Horizon Research Agents." 2026. Arxiv 2610.02525
AuthorsOpenReview ↗

Follow-up to Reviewer ZRg7

Dear Reviewer ZRg7,

Thank you again for your review. For your three requests we ran these experiments; all are in the revised paper.

  • A table as the main result (Q1). Table 1 gives five paired comparisons, and every system on every task is in the appendix.
  • A baseline that keeps its goal (Q2). Codex /goal, with a persistent goal, declares 197 of 200 tasks complete after a median of 36 minutes (Section 2).
  • A hierarchical baseline (Q3). Recursive Language Models on all 200 tasks: ProContract is +7.73 points higher [5.84, 9.75] (Table 1; Appendix C.5).

Do these answer your requests? If anything remains open, or there is another baseline you would like to see, please tell us; we will answer promptly.

With thanks,

The authors

Reviewer_gQB2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors note that llm-based agents are unreliable because they can rewrite or delete the task they have been given, and because they judge themselves whether they have solved it or not. They propose separating those pieces of information from the "working memory" of the agent. Some experiments were done.

Critical strengths

According to the authors, this rather obvious deficiency of recent agents has not been addressed previously.

Critical weaknesses

The paper is very vague on details. It is never specified what form "task requirements" can take, how the "verifier" works, what "evidence" is, and so on. The paper refers to several propositions/theorems none of which appear in the paper. The only "formal" statement in the paper is "invariant 1", which is unclear (and, of course, not proven): it is formulated using a number of terms that are not defined, or even casually explained, anywhere else ("K", "authenticated actors", "immutable identities"), and its use of notation is inconsistent with what was introduced earlier. Where "Live(a)" is "defined", "a" is an acceptance, which is described as being composed of "the requirements [...], the frozen artefact, its evidence, the verifiers judgement, and the acceptances it relies on", while the invariant uses "Live(sigma_tau)", where sigma_tau is a "delivery", a term that is also undefined, but which I believe is different from an acceptance since the authors make the point that a "contract" separately stores a task's acceptances and its "current delivery".

The related work section is missing an overview of the earlier generation of agent systems (blackboard systems, the subsumption architecture, logical agent languages, and so on) and how those addressed this problem (if it was indeed ever a problem).

Questions for authors

None.

8 comments
AuthorsOpenReview ↗

Response to Reviewer gQB2 (1/2)

Dear reviewer gQB2

Thank you for recognizing the deficiency the paper addresses, which your summary states plainly: an agent can rewrite or delete the task it has been given, and it judges for itself whether it has solved it. We answer in two comments: what the paper adds and the definitions here, earlier agent architectures in the second. Each answer stands on its own; the passages behind it are set in blue in the revised paper.

1. What the paper adds

Your comments: "this rather obvious deficiency of recent agents has not been addressed previously"; informativeness is rated 1. TL;DR: The principle is simple, and it is old. The paper adds three things: evidence that production agents lack it, a measurement of why, and a mechanism that enforces it while the agent's own code is replaced or rewritten.


  • Evidence. A production coding agent, in its mode built for long tasks, declares 197 of 200 six-hour tasks complete after a median of 36 minutes. One program passes every hidden test.
  • Why. The agent has not overlooked the gap: in 62 of the 197 completion messages it names what is missing and closes the task in the same message. It is not a lack of ability either: when its "done" is refused with one fixed sentence, the same agent finds and fixes failures for a median of 5.3 more hours, and in 5 of 32 such runs it rewrites its goal into a narrower one. The labs that build these agents report the same behavior in their own systems (general response, part 1).
  • A mechanism. Three properties hold together: the harness may change and rewrite itself; the Contract (what is owed, what has been accepted, who may judge) is the same under every harness version; and each acceptance stays bound to its frozen artifact, its authorized judgment and its dependencies, and is withdrawn when a basis fails. Earlier systems have parts of this (our second comment); among those we compare, none combines the three. ProContract scores 5.07 points higher than the same executable without it, and the resulting system has the highest mean pass rate on the public ProgramBench leaderboard.

In the revised paper: Section 2, lines 134–178; Section 4.3, lines 429–445; Table 1.

2. Definitions and the formal statement

Concern (summary): The paper never specifies the requirements, the verifier, the evidence or the delivery; the invariant uses undefined terms and is not proven; the propositions it refers to do not appear. TL;DR: The propositions, the theorem and the proof of the invariant are in Appendix A of the submission, and the main text now says so where the first one is used. Each term is now instantiated on the benchmark at the point where it is introduced, and the delivery is defined as one of the task's acceptances, so the invariant is well typed.


TermMeaningOn ProgramBench
Requirementswhat the artifact must satisfy; only the user can change themthe task statement: rebuild the reference program
Artifactthe frozen work that is judgeda frozen copy of the workspace
Evidencewhat the judgment rests onthe recorded runs of that copy against the reference program
Verifierthe role whose judgment countsa reviewer role that the worker cannot assume
Acceptance aa positive judgment recorded by the kernel, bound to the requirements, the artifact, the evidence and the acceptances it relies onone record for an accepted candidate
Delivery σthe acceptance whose artifact is delivered; none while the task is outstanding
Live(a)the acceptance still stands: it is not withdrawn, its requirements are current, an authorized verifier issued it on evidence that meets the rule, and every acceptance it relies on is live

The idea in one line. An ordinary agent has one state that its harness rewrites: s' = H(s). ProContract splits it: s' = H(s, C) and C' = K(C, r). C is the set of Contracts and K is the kernel, a deterministic program that runs no model. The harness H can read C and send requests r.

The invariant. If a task is fulfilled, then σ ∈ A and Live(σ): a fulfilled task always has a current delivery that still stands. It rests on three assumptions, now stated in plain words: every write to a Contract passes through K as one atomic step, K knows which role issued each request, and a named artifact or piece of evidence cannot be altered afterwards. The proof concerns the kernel's transitions only, so it holds under any harness.

In the revised paper: Equation 1 and lines 204–209; Section 3.1, lines 236–238 and 254–293; proofs in Appendix A.3 and A.4.

AuthorsOpenReview ↗

Response to Reviewer gQB2 (2/2)

3. Earlier agent architectures

Concern (summary): The related work omits the earlier generation of agent systems (blackboard systems, the subsumption architecture, logical agent languages) and how they addressed this problem. TL;DR: The related work now covers them and says how each handled the problem. ProContract builds on three of them: commitments, truth maintenance and the separation of duties. It makes them a boundary that holds while the harness itself is replaced or rewritten.


Earlier workHow it treated the task and its completionWhat ProContract takes or changes
Blackboard systems, such as Hearsay-II [1]problem state on a shared blackboard, apart from the knowledge sources; every source may write ittask state apart from the worker; only a kernel writes it
Subsumption [2]layered control with no shared record of the taska record, because the record is what must survive a change of method
Logics of intention; AgentSpeak(L) [3, 5]an agent keeps a goal until it believes the goal achieved or impossiblethat belief is what we measure: a language-model agent believes it too early
Social commitments [4]the obligation is public, and is discharged by events that others can recognizethe Contract: discharged by a verifier's judgment, not by the worker
Truth maintenance [6, 7]a conclusion is kept only while its support standsan acceptance is withdrawn when its basis fails
Commercial integrity policies [8]the party that performs a transaction does not certify itthe worker does not record its own progress

A judgment binds to one frozen artifact and to the current requirements, and it counts only while its support holds. No change to the harness can rewrite the record, because every write passes through a deterministic kernel. The same boundary governs the adoption of a revised harness, so the agent can rewrite how it works in the middle of a task without changing what it owes.

In the revised paper: Section 5, lines 509–514; Appendices A.7 and D.3.

References:

  1. Erman, Hayes-Roth, Lesser and Reddy. "The Hearsay-II Speech-Understanding System: Integrating Knowledge to Resolve Uncertainty." ACM Computing Surveys, 1980.
  2. Brooks. "A Robust Layered Control System for a Mobile Robot." IEEE Journal on Robotics and Automation, 1986.
  3. Cohen and Levesque. "Intention Is Choice with Commitment." Artificial Intelligence, 1990.
  4. Singh. "An Ontology for Commitments in Multiagent Systems: Toward a Unification of Normative Concepts." Artificial Intelligence and Law, 1999.
  5. Rao. "AgentSpeak(L): BDI Agents Speak Out in a Logical Computable Language." MAAMAW, 1996.
  6. Doyle. "A Truth Maintenance System." Artificial Intelligence, 1979.
  7. de Kleer. "An Assumption-Based TMS." Artificial Intelligence, 1986.
  8. Clark and Wilson. "A Comparison of Commercial and Military Computer Security Policies." IEEE Symposium on Security and Privacy, 1987.
AuthorsOpenReview ↗

Follow-up to Reviewer gQB2

Dear Reviewer gQB2,

Thank you again for your review. Here is what we did for your three concerns; all of it is in the revised paper.

  • What the paper adds (Q1). Three properties hold together, and to our knowledge no earlier system combines them: a harness that can rewrite itself, a Contract that no harness version can alter, and acceptances that stay bound to their artifact and judgment (Appendix D).
  • Definitions and the formal statement (Q2). Each term is now instantiated on the benchmark (Section 3.1). We also ran the formal property on a real task: we replaced the harness under a running task, and the Contract, its history and its allowance are identical before and after (Section 4.4; Appendix C.6).
  • Earlier agent architectures (Q3). The related work is extended (Appendix D).
  • The main result with the executable fixed. ProContract switched off on the same executable, 200 tasks: +5.07 points [3.70, 6.54] (Table 1).

Do these answer your concerns? If anything remains open, please tell us; we will answer promptly.

With thanks, The authors

Reviewer_z61DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that long-horizon agents lose track of requirements and misjudge progress because both live in mutable harness state. ProContract moves them into Contracts held by a deterministic kernel outside the agent: the harness can read a Contract but can only complete a task by submitting a frozen artifact that a designated verifier accepts. Because task state lives outside the harness, the harness can be replaced mid-task, and improving the harness becomes another verified task, enabling recursive self-improvement in which a generated harness writes its successor. Built on OpenCode, ProContract reports a ProgramBench mean pass rate of 77.24% with GPT-5.6 Terra versus 72.35% for a public mini-SWE-agent run, 75.25% with GPT-5.6 Sol versus 69.89% on the official leaderboard, and 65.7% on Terminal-Bench 2.0 with GPT-5.2.

Critical strengths

The paired gain over the same model's public run is 5.33 points (95% CI 2.90–7.95) with Terra and 5.36 (2.46–8.39) with Sol, concentrated on the hardest tasks (tasks below 10% halve, Section 4.2), and with Sol ProContract is also cheaper per task than the official run ($3.14 vs $6.08, Table 5).

Critical weaknesses
  • ProContract is implemented on OpenCode (Appendix C.1.1), while every baseline is a public mini-SWE-agent run (Table 5). There is no run of plain OpenCode without the kernel, so harness differences and the kernel are confounded and the gain cannot be attributed to the kernel.
  • The reported ProgramBench results are single runs selected from eleven 200-task runs by full-solve count, then mean score (Appendix C.1.1). This is not mentioned in the main text, and the number of runs per model and the scores of unselected runs are not reported. The confidence intervals resample tasks within the selected run and do not capture run-to-run variance, whereas Terminal-Bench results average two trials (Appendix C.3). The reported gains may therefore be optimistic.
  • The evidence does not show the kernel doing what the paper claims. The mechanism the paper credits, continuing past a passing self-check, accounts for only about +1.1 points (Section 4.2) of the 5.3-point gain, and the kernel's withdrawal and dependency machinery never activates: no withdrawals, releases or prerequisites occur across all 200 tasks (Appendix C.1.2). Moreover, on ProgramBench the verifier only rebuilds the code and reruns regression tests written by the agent itself (Section 4.1), so progress is still certified by the agent's own evidence rather than an independent judgment. The gain may come from forcing the agent to keep working rather than from the kernel.
  • The recursive self-improvement claim is not supported. The self-improved harness v2 scores 2.16 points below the base harness on paired tasks (Section 4.3), so the experiments show that the agent can rewrite its harness without breaking its contracts, not that self-improvement improves performance.
Questions for authors
  1. How does plain OpenCode, with the same model, prompts and limits but without the kernel, score on the same 200 tasks?
  2. Can you report all eleven runs: how many per model, which run produced the base Terra result (77.24%), and the mean and spread per model?
  3. Did any withdrawal, release or dependency event occur in any reported experiment? If not, which result depends on the dependency tracking?
  4. Does any self-improved harness version beat the base on the full benchmark?
5 comments
AuthorsOpenReview ↗

Response to Reviewer z61D (1/3)

Dear reviewer z61D

Thank you for recognizing that the paired gain over the same model's public run holds with two models (5.33 and 5.36 points), that it is concentrated on the hardest tasks, and that with Sol it comes at a lower cost per task. The stricter experiments you ask for reproduce it. Your four questions are precise: the first two are answered here, the third and the fourth in two further comments. Each answer stands on its own; the passages behind it are set in blue in the revised paper.

1. Is the gain due to the kernel or to the harness?

Q1 (summary): ProContract is built on OpenCode and every baseline is a public mini-SWE-agent run. How does the same harness score without the kernel, with the same model, prompts and limits, on the same 200 tasks? TL;DR: On one executable, with the same model and the same six hours, the system scores 71.22% with ProContract switched off and 76.28% with it switched on: a gain of 5.07 points (95% interval 3.70 to 6.54). The submission's comparison with the public run of the same model gave 5.33, so holding the executable fixed does not change the result.


Why the submission compared with public runs. A public leaderboard run is produced by a third party, so we cannot tune it, and it holds the model fixed. It does not hold the executable fixed, which is what you ask for.

What "switched off" is. We now build ProContract on Codex, as an extension of its runtime. Off is the same binary with ProContract disabled and Codex's own mode for long tasks, /goal, enabled: the agent keeps a persistent goal, continues automatically, and marks the task complete itself. We ran it twice, on our build and on the stock release. Switched on is ProContract as a whole: the kernel, the verifier, and the harness that evolved under them. Answers 3 and 4 take that whole apart.

System (GPT-6 Luna, 200 tasks, six hours)Mean pass rateGain of ProContract [95% interval]Tasks higher / lowerTasks below 50%ResolvedRequests per taskHours used
Codex /goal, same executable71.22%+5.07 [3.70, 6.54]154 / 403701560.8
Codex /goal, stock release71.20%+5.09 [3.78, 6.45]158 / 373611570.7
ProContract76.28%2431,4135.9

Both systems had the same six hours and no limit on model requests. The baseline used 0.8 hours because it declared its task done.

In the revised paper: Table 1; Section 4.1, lines 360–364; Section 4.3, lines 396–398.

2. Selection among runs, and run-to-run variance

Q2 (summary): The reported results are single runs selected from eleven 200-task runs. Can you report all eleven: how many per model, which run produced the base Terra result (77.24%), and the mean and spread per model? TL;DR: The eleven were development runs of successive harness configurations, three, six and two per model, and the base Terra result is the only base run with that model. The revision uses none of them: every result is one cohort of the 200 tasks, run once, with its model, effort and limits fixed before it started. The gain does not depend on the run: the baseline run twice gives 71.22% and 71.20%, and three ProContract cohorts against five baselines give gains between 4.45 and 7.91 points.


What was repeatedRunsResult
Codex /goal on all 200 tasks2, on two builds71.22% and 71.20%; one task differs by 5.8 points on average
ProContract on 25 tasks274.3% and 73.4%
ProContract on Terminal-Bench 2.0, 89 tasks362, 57 and 57 tasks passed
The gain of ProContract, across comparisons5 in the revision, 2 in the submission+5.07, +5.09, +7.73, +4.45, +7.91; in the submission +5.33 and +5.36

The eleven reports of the submission. They were development runs of successive harness configurations, not repeats of one: three with GPT-5.6 Luna (mean 69.3%; 67.5% to 70.2%), six with GPT-5.6 Sol (mean 73.1%; 65.4% to 81.1%; the reported 75.25% is at xhigh, the effort of the public run it is compared with) and two with GPT-5.6 Terra (the base, the 77.24% you ask about, and its generated successor v2 at 75.72%). The revision uses none of them: every cohort in Table 1 is run once under a protocol fixed in advance.

Single tasks vary between runs, which is why every comparison is paired over 200 tasks and reports an interval over tasks. An attempt that a provider or host failure ended was run again as a fresh attempt, and the repeat is the one scored, whatever either attempt scored.

In the revised paper: Table 1; Appendices C.1, C.2 and C.11.

AuthorsOpenReview ↗

Response to Reviewer z61D (2/3)

3. What the kernel does

Q3 (summary): Continuing past a passing self-check accounts for +1.1 of 5.3 points; withdrawal and dependencies never activate; the verifier reruns the agent's own tests. Does the gain come from forcing the agent to keep working, not from the kernel? TL;DR: Keeping the agent at work after it says "done" is the kernel's rule: the worker's "done" does not close the task. We tested that rule alone, as a loop with no kernel and no verifier, and it recovers the gain: higher than plain /goal on 26 of 30 tasks. What the kernel and the verifier add is that the task cannot be rewritten (0 of 200 runs, against 5 of 32 under the loop), that it survives a change of worker and of harness version, and that it can close on a judgment that is not the worker's.


Codex /goal/goal, every "done" refused by a loopProContract
Pass rate against plain /goalbaselinehigher on 26 of 30 tasks, median +5.4 pointshigher on 154 of 200 tasks, mean +5.07
Who ends the taskthe worker, after a median of 36 minutesno one: no run can closeonly the verifier's acceptance
Task rewritten by the worker–in 5 of 32 runsin 0 of 200 runs
If the worker's session ends–the task ends with it: 1 run lost at a context overflow (0%, against 93.6% under plain /goal)a new worker continues the same task, 4 times per task (median)

The +1.1 points. That figure compared two artifacts inside one ProContract run: the first whose own check passed, and the one finally delivered. It measures only the part of the gain that comes after the first passing check inside a ProContract run. The loop measures the effect from outside: the worker marks one task complete 429 times in 32 runs, and in the first four hours 85% of these marks are followed by a new commit of its own. Each mark is what we call progress misjudgment.

The verifier. The kernel's guarantee does not depend on how strong the verifier is: it binds each judgment to one frozen artifact and to the current requirements. The submission's verifier, which rebuilt the artifact and reran the worker's regression suite, was the simplest one that exercises this binding. The revision uses a stronger one, a separate role that the worker cannot assume and that gathers evidence of its own, and the kernel's rules did not change. On Terminal-Bench 2.0 it accepts 89 of 267 attempts, at a median of 52% of the time limit, and 84% of them pass the benchmark, against 57% of the attempts that run to the limit.

Withdrawal and dependencies. They keep an acceptance from outliving its basis. A benchmark task is a single Contract with no prerequisite, so they have nothing to act on there. They acted where there was something: an audit found that the author of two adopted harness revisions could see the evaluation sources, and Withdraw reopened both adoptions and kept their history.

In the revised paper: Section 4.3, lines 429–445, and Figure 6d; Section 3.1, lines 236–238 and 287–289; Section 4.4, lines 498–499; Appendix C.11.

AuthorsOpenReview ↗

Response to Reviewer z61D (3/3)

4. Does self-improvement improve performance?

Q4 (summary): The self-improved harness v2 scores 2.16 points below the base. Does any self-improved harness version beat the base on the full benchmark? TL;DR: Yes, and we report two measurements that should be read apart. With the kernel on and the model and the tasks held fixed, one harness revision takes early stops from 34 of 37 runs to 4. The whole evolved system, the kernel with the harness after 23 released revisions, scores 5.07 points above Codex, the base it started from, on the full benchmark. v2 is something else: a candidate that the adoption rule rejected, so it never became the system.


What is comparedWhat is held fixedResult
One revision: the harness before and after it waits through provider limitskernel on; GPT-6.1 Sol; the same 37 tasksearly stops fall from 34 of 37 runs (after a median of 0.3 hours) to 4; the median run then uses the full six hours
The evolved system (kernel and harness after 23 revisions) against Codex, its baseexecutable; GPT-6 Luna; six hours; 200 tasks+5.07 points [3.70, 6.54]
v2 against its base, in our early exploration (not adopted)kernel on; GPT-5.6 Terra; 200 tasks−2.16 points, at about half the cost per task ($7.22 against $13.50)

Across the three successive cohorts (GPT-5.6 Luna, GPT-6.1 Sol, GPT-6 Luna) early stops fall from 47% to 36% to 4% of runs; the first row is the same effect with the model fixed.

One step of self-improvement, end to end. Each of the 23 revisions is one pass of the same loop. Three of them:

Link of the loopA tool (revision 4)The improver itself (revision 5)Execution (revision 6)
Failure observeda tool returns a wrong sum in the model's own taskthe validator of research tables accepts 0.63 written as 1reviews are blocked after a failed snapshot, in the GPT-6.1 Sol runs
Revision written bythe model, inside an improvement taskthe research agenta second agent, from the research agent's diagnosis
Check6 of 6 fresh private casesthe same bytes: the old version accepts, the new one rejectsthe old harness fails the control, the new one passes
Adoptiona Selection Contract, proposed by the modelreview by the user and a second modelthe user, at release
Later usethe next task calls the toolit checks the next reportin the next cohort such stops fall from 28 runs to 5

The user sets the objectives and authorizes adoptions, which the design keeps outside the agent; models and tools do the analysis, the revisions and the checks. The loop is recursive because the improver is part of the harness it improves: revision 5 changed how later revisions are judged.

What v2 shows. The −2.16 is the adoption rule doing its job. The agent generated a version that looked 6.4 points better on the tasks it was developed on. The rule, which no version can modify, asked for the gain on tasks that version had not seen, found −0.2, and kept the base. A gain that was noise never reached a user.

Why revisions are judged by regression checks and not by a benchmark score. Two runs of the same baseline differ by 5.8 points per task on average, so telling a two-point revision from noise would take more than 200 paired six-hour runs. A regression check takes minutes: one input, taken from the run where the harness failed, that the old version fails and the new one must pass.

In the revised paper: Section 4.4, lines 451–453, 473–485 and 487–494; Section 3.2, lines 323–350; Figures 1c and 7; Appendix C.6, Tables 9 and 10; Appendix C.7.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response (1/3) on the problem, and what the new experiments confirm

Dear Area Chair and all reviewers:

We thank the reviewers for recognizing the two named failure modes and the decoupling of responsibility and progress from the harness (Gpd9), the formalization of responsibilities, the deterministic kernel and its support for self-improvement (ZRg7), the paired gain concentrated on the hardest tasks (z61D), and the deficiency itself (gQB2).

TL;DR: Agents are now given tasks that take hours, and the agent itself decides when the task is done. The submission showed that taking this decision away from the agent lets the same model deliver more. The reviews ask for tighter controls and stronger baselines. We ran every one of them, and each reproduces the submission's result: with the executable also held fixed, ProContract scores 5.07 points above Codex (5.33 in the submission, against the public run), and it sets the state of the art on ProgramBench, 89.70%.


The problem. A long task ends when someone says it is done. Today that is the worker, and it stops as soon as it knows of no failure. In progress misjudgment it takes a passing check of its own for completion. In responsibility dilution it narrows the task to what its method covers. Both have one cause: the task is held by the method that works on it, the agent's harness. That has a second cost: a harness has to change, for every new model and increasingly by the agent itself, and while it holds the task each new version can also change what is owed and what counts as done. ProContract takes the task out of the harness: a small deterministic kernel holds it, and progress counts only when an independent verifier accepts the exact work submitted. Three things then hold together, and the combination is what sets ProContract apart: (1) a harness that can change and rewrite itself; (2) a Contract that no harness version can alter: what is owed, what has been accepted, and who may judge; (3) acceptances that stay bound to their frozen artifact, their authorized judgment and their dependencies, and are withdrawn when a basis fails, whichever version runs. Among the systems we compare (Appendix D), none combines the three.

Why it matters. Agent work is getting longer: OpenAI reports a coding agent that works on one task for more than 24 hours [1], and Anthropic a "tendency to mark a feature as complete without proper testing" [2]. Google DeepMind argues that delegation requires an explicit transfer of authority, responsibility and accountability [3]. The harness itself does not stand still: every component "encodes an assumption about what the model can't do on its own", and these "can quickly go stale as models improve" [4]; Meta builds agents that rewrite even the procedure by which they improve themselves [5]. When an agent can rewrite its context, its harness and its own improvement process, the task and the judgment that it is complete have to be kept where no version of the agent can rewrite them.

What the submission showed, and what the new experiments show.

ClaimIn the submissionIn the new experiments
The same model delivers more+5.33 and +5.36 points over the public run of the same model (two models)+5.07 on the same executable; +4.45 and +7.91 over the public runs of two more models; +7.73 over Recursive Language Models
The gain is at the bottom of the distributiontasks below 10% fall from 15 to 3tasks below 50% fall from 37 to 24; the 10th percentile rises by 10.2 points
The mechanism: the task stays open after the worker's own check passeswork after the first passing self-check adds 1.1 points within a runrefusing the worker's "done", and nothing else, raises the score on 26 of 30 tasks
The harness evolves while the task holdsa generated harness produced its successor with every Contract intact23 released revisions; with the model and the tasks fixed, one takes early stops from 34 of 37 runs to 4
It holds on a second benchmarkTerminal-Bench 2.0: 65.7%65.9% on another runtime; the public harnesses for that model pass 54.0% to 64.9%

Cited passages are set in blue in the revised paper; line numbers are those in its margin.

References:

  1. OpenAI. "Building more with GPT-5.1-Codex-Max." 2025.
  2. Young. "Effective harnesses for long-running agents." Anthropic, 2025.
  3. Tomašev et al. "Intelligent AI Delegation." Google DeepMind, 2026. Arxiv 2602.11865
  4. Rajasekaran. "Harness design for long-running apps." Anthropic, 2026.
  5. Zhang et al. "Hyperagents." Meta, 2026. Arxiv 2603.19461
AuthorsOpenReview ↗

General response (2/3) on what each review asked for, and the answer

TL;DR: Each request is answered by a new experiment, or by text now in the paper, and each answer is given here in full. The experiments are new. The two failure modes, the kernel, its formal model and its proofs are those of the submission.


Asked forByAnswerIn the revised paper
ProContract switched off on the same executablez61D Q1, Gpd9 Q1+5.07 points [3.70, 6.54] on 200 tasks; higher on 154, lower on 40Table 1; Section 4.3, lines 396–398
The eleven runs of the submission; run-to-run variationz61D Q2the eleven were development configurations (three, six and two per model), and the revision uses none; one cohort per result, run once; the baseline run twice gives 71.22% and 71.20%Table 1; Appendices C.1 and C.2
Is the gain only forced continuation? A feedback-matched baselinez61D Q3, Gpd9 Q3keeping the task open carries the gain, with no feedback at all (26 of 30 tasks); only ProContract keeps the task from being rewritten (part 3/3)Section 4.3, lines 429–445
A baseline that keeps its goalZRg7Codex /goal, with a persistent goal, declares 197 of 200 tasks complete after a median of 36 minutesSection 2, lines 136–138
A hierarchical baselineZRg7Recursive Language Models on all 200 tasks: ProContract is +7.73 higher [5.84, 9.75]Table 1; Appendix C.5
Matched effort; the complete leaderboardGpd9 Q2every run at max effort; 89.70% against 87.79% for the best of 66 public entriesSection 4.2, lines 371–376
A self-improved harness above its basez61D Q4the system we report is that harness: +5.07 above Codex, its base, after 23 released revisions, each a full step from observed failure to later useSection 4.4, lines 473–485
A table as the main resultZRg7Table 1: five paired comparisons; every system on every task in the appendixTable 1; Appendices C.9 and C.10
Definitions, the invariant, its proof; earlier agent architecturesgQB2each term instantiated on the benchmark; the proofs are in Appendix A, as in the submission; related work extendedSection 3.1, lines 236–238 and 254–256; Appendix D.3

What changed, and why. We now build ProContract on Codex. Codex ships a mode built for long tasks, /goal, so switching ProContract off on the same executable leaves a production baseline, not a bare agent loop. All main experiments are new, with GPT-6 Luna, GPT-6.1 Sol and GPT-5.6 Luna. The two implementations agree where both ran: on Terminal-Bench 2.0 the submission's implementation passes 65.7% of the tasks and the new one 65.9%.

AuthorsOpenReview ↗

General response (3/3) on task quality and task integrity

TL;DR: Four cells on one panel of 40 tasks separate two outcomes: the quality of the result, and the integrity of the task. Keeping the task open recovers the quality, with or without ProContract. Only ProContract keeps the task itself: it is never rewritten, it survives a change of worker, and it closes on a judgment other than the worker's. Continuing work recovers capability; a protected Contract makes that continuation accountable.


Cell (GPT-6 Luna, 40 tasks)What ends an attemptWhat we observe
Codex /goal, plainthe worker's "done"the worker stops after a median of 0.5 hours
Codex /goal, every "done" answered with one fixed sentencethe time limit; no run can close its taskhigher than plain on 26 of 30 tasks (median +5.4 points); the worker rewrites its goal in 5 of 32 runs
ProContract, with the worker allowed to end after the verifier's assessmentthe worker; a turn's end only freezes a candidate for the verifier5.5 points above /goal (95% interval 2.3 to 9.1); shown a counterexample, the worker takes back its "finish" 47 times out of 49; it still ends 18 of 25 attempts, and none of the 18 programs passes every hidden test
ProContractthe verifier's acceptance, or the time limit7.1 points above /goal (2.2 to 12.9); the worker cannot end the task; this second run of ProContract agrees with the first (74.3% and 73.4% on these tasks)

(Section 4.3, lines 429–445, Figure 6c and 6d; Appendix C.11, Tables 21 to 24 and Figure 11.)

  • Quality: the open task carries the gain. Refusing "done", with no verifier and no feedback, scores higher than plain /goal on 26 of 30 tasks (median +5.4 points): the gain comes from keeping the task open, not from feedback.
  • Progress misjudgment: "done" is a pause, not a judgment. Refused, the worker marks the same task complete 429 times in 32 runs. In the first four hours, 85% of these marks are followed by a new commit of its own.
  • Responsibility dilution: a loop cannot hold the task. In 5 of the 32 runs the worker writes itself a narrower goal, one run loses its program to a context overflow, and no run can close its task, because nothing there can say that the task is met. ProgramBench scores whatever program exists at six hours, so a loop that never stops loses nothing there; a task that has to be delivered, closed or handed on needs someone who can say that it is met.
  • Integrity: a kernel and a verifier hold the task and let it close. In the 200-task run no task is rewritten, and the worker is replaced a median of 4 times per task with the task unchanged. Shown a counterexample on its exact candidate, the worker takes back its own "finish" 47 times out of 49. On Terminal-Bench 2.0 the verifier accepts 89 of 267 attempts, after a median of 52% of the time limit, and 84% of them pass the benchmark, against 57% of the attempts that run to the limit (Section 4.3; Appendices C.4, C.6 and C.8).

The alternatives raised by the reviewers differ, and they fail in the same place: with a persistent goal, a loop that forces work, or a root model in charge, the system that does the work still says when the work is finished, or can rewrite what is to be finished.

Why a loop is not enough: workers and harnesses do not last. A loop that refuses "done" holds the task only while one worker and one harness last. In the 200-task run the worker is replaced a median of 4 times per task, and the harness itself went through 23 revisions. The kernel's guarantees hold across both, so the harness can evolve while the task holds, and an open task shows what to revise: a harness can still stop early, but the run then records that it stopped and why. With the model and the tasks held fixed, one revision takes early stops from 34 of 37 runs to 4. Across three successive cohorts (GPT-5.6 Luna, GPT-6.1 Sol, GPT-6 Luna) the share falls from 47% to 36% to 4% (Section 4.4, lines 451–453, Figure 7; Appendix C.6).

AuthorsOpenReview ↗

Follow-up to the Area Chair and all reviewers: the experiments we ran, and an invitation to discuss

Dear Area Chair and Reviewers,

Thank you again for your reviews and for the time you have given this paper. To make the discussion easy to continue, this note lists in one place the experiments we ran in response to the reviews, and where each one is in the revised paper (changes are set in blue).

TL;DR: We ran every control and baseline the reviews asked for, and each reproduces the submission's result. If any of them leaves a question open, we would be glad to discuss it.


ExperimentForResultIn the revised paper
ProContract switched off on the same executable, 200 tasksz61D, Gpd9+5.07 points [3.70, 6.54]; higher on 154 tasks, lower on 40Table 1; Section 4.3
The baseline run twicez61D71.22% and 71.20%Table 1
Four cells on one panel of 40 tasks, from plain /goal to ProContractz61D, Gpd9refusing the worker's "done", and nothing else, scores higher on 26 of 30 tasks (median +5.4 points); shown a counterexample, the worker takes back its "finished" 47 times out of 49Section 4.3; Appendix C.11
A baseline that keeps its goalZRg7Codex /goal declares 197 of 200 tasks complete after a median of 36 minutesSection 2
A hierarchical baseline: Recursive Language Models on all 200 tasksZRg7ProContract is +7.73 points higher [5.84, 9.75]Table 1; Appendix C.5
More models, max effort everywhere, the complete leaderboardGpd9+4.45 to +7.91 points over the same model; 89.70% mean pass rate against 87.79% for the best of 66 public entriesTable 1; Section 4.2
A second benchmark, with its reference harness as the controlallTerminal-Bench 2.0: 65.9% and 65.7% on two runtimes; 57 of 82 tasks against 42 for the reference harness with the same modelSection 4.2
Self-improvement: two rounds, 23 released revisions of the harnessz61Druns that stop early at a defect of the harness fall from 40% to 18% to 4%Section 4.4; Appendix C.6
The harness replaced under a running taskallthe Contract, its history and its allowance are identical before and after; six more submissions and two Defeats follow, each naming its own copySection 4.4; Appendix C.6

Do you have further questions, or a control, a baseline or an analysis you would still like to see? Please tell us. We will answer promptly and run what can be run in the time that remains.

With thanks, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.