- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Long-horizon agents turn user-assigned responsibility into artifacts over hours of execution, but they often lose track of the responsibility because it and its progress are entangled with the execution state the agent continuously rewrites, producing two failure modes the paper calls responsibility dilution and progress misjudgment. ProContract introduces a progress-contract kernel that keeps responsibilities in a Contract outside the agent and records progress only through authorized judgments, so changes in execution state cannot alter assigned responsibilities or interfere with progress judgment. The agent's harness is deliberately disentangled from the kernel, which also makes self-improvement an ordinary responsibility: the agent can rewrite its own harness while the kernel preserves the task it was asked to carry out.
Critical strengths
- The paper names two long-horizon failure modes and decouples responsibility and progress from the harness to address them. It identifies responsibility dilution and progress misjudgment as failures that arise because the responsibility and its progress are entangled with the execution state the agent continuously rewrites. ProContract keeps each responsibility in a Contract that only the kernel can change, has the harness read the Contract but never record progress itself, and admits progress only through a verifier's authorized judgment, so changes in execution state cannot alter assigned responsibilities or interfere with progress judgment.
- Progress is recorded only through a careful per-acceptance verifier rule. Progress enters only as a verifier's judgment on a frozen artifact, recorded by the kernel as an acceptance, and the kernel checks the judgment against the state in which it takes effect, so a diagnostic check run by the harness is not progress, a judgment made under stale requirements is rejected, and a judgment of one artifact never transfers to another.
Critical weaknesses
- Every baseline, including the official leaderboard entries and the Terra comparison, runs mini-SWE-agent, a minimal harness, while ProContract is a full system built on OpenCode with structured tool results, observation packing, diagnostic compaction, and a controller, and it spends 2.5 times as much per task. The comparison therefore varies the harness implementation, the tool surface, the context-management strategy, and the cost at the same time as the kernel, so it is not an exact comparison that isolates the kernel.
- The headline comparison depends on which reasoning-effort setting the baseline uses. The main gain is reported against GPT-5.6 Sol at xhigh effort, which scores 69.89, but Table 6, which the paper calls the complete third-party comparison population, reports the same model at max effort running plain mini-SWE-agent at 77.64, above ProContract's 75.25. The claim that ProContract Sol has the highest mean pass rate holds only for the Table 5 population, and the main text uses the lower-scoring baseline setting without addressing the higher one.
Questions for authors
- Mini-SWE-agent differs from ProContract in the harness implementation, the tool surface, the context-management strategy, and the cost, not only in the kernel. Please run ProContract's own OpenCode harness with the kernel disabled, keeping the same runtime, tools, observation packing, diagnostic compaction, and cost envelope, but letting the harness mark its own tasks done, and report the 200-task distribution. This is the only comparison that isolates the kernel from the harness implementation, and the framework already supports it.
- Please add GPT-5.6 Sol at xhigh effort to Table 6. The headline gain is reported against Sol at xhigh (69.89), but Table 6 reports Sol at max effort running plain mini-SWE-agent at 77.64, above ProContract's 75.25, and the two tables use different effort settings for the same model. Please report Sol at xhigh within the Table 6 population as well, so the baseline is matched in reasoning effort, and state whether the claim that ProContract Sol has the highest mean pass rate still holds under the complete third-party population.
- Please report whether the comparison is fair given that ProContract receives per-subtask verifier feedback. In the default setting a model solves the task on its own, whereas ProContract submits artifacts to a verifier and receives accept or withdraw judgments during execution, which mini-SWE-agent does not. A higher pass rate under this additional feedback would be expected regardless of the kernel, so please run a feedback-matched baseline that gives the baseline harness the same submit-and-judge interactions and the same accept or withdraw signals but without the kernel's responsibility-progress decoupling, and report how much of the gain remains.
8 comments
Response to Reviewer Gpd9 (2/2)
3. Verifier feedback
Q3 (summary): ProContract receives verifier feedback during execution that the baseline does not. With a feedback-matched baseline, how much of the gain remains?
TL;DR: All of it remains with no feedback at all: when the worker's "done" is answered by one fixed sentence, with no verifier and no information about its work, the score is higher than plain /goal on 26 of 30 tasks (median +5.4 points). The gain therefore comes from the task staying open, not from the verifier's feedback. With the verifier's assessments given in both arms, the arm in which the worker may still end the task scores 5.5 points above /goal, and the arm in which it may not scores 7.1 above; the 1.6 points between them (95% interval −2.4 to 6.6, 25 tasks) are what taking the end away from the worker adds.
Why the verifier is part of the system and not an advantage in information. The verifier sees no hidden test. It runs the reference program, which the worker also has, on the worker's frozen candidate. The baseline makes the same kind of observation itself: 183 of its 197 completion messages cite its own comparison with the reference.
The four systems (GPT-6 Luna, one panel of 40 tasks):
| System | Feedback to the worker | Who may end the task | Result |
|---|---|---|---|
Codex /goal | none | the worker | stops after a median of 0.5 hours |
/goal, every "done" refused | one fixed sentence: "Continue working on the same goal" | no one | higher than plain /goal on 26 of 30 tasks, median +5.4 points |
| ProContract, the worker free to end | the verifier's assessment | the worker, after an assessment | 5.5 points above /goal (2.3 to 9.1); the worker ends 18 of 25 attempts |
| ProContract | the verifier's assessment | only the verifier | 7.1 points above /goal (2.2 to 12.9); 1.6 above the row before (−2.4 to 6.6) |
The third row already runs under the kernel. There a turn's end does not close the task, as it does in /goal: the kernel freezes the candidate and the verifier assesses it. That is why the row keeps the gain. What it lacks is the last step: the worker ends 18 of the 25 attempts, at a median of 4.4 hours, and none of the 18 programs passes every hidden test.
What the kernel and the verifier add is integrity: the task stays the user's task. Under the loop the worker rewrites its goal in 5 of 32 runs, which is what we call responsibility dilution, and no run can close. Under ProContract no task is rewritten in 200 runs. On Terminal-Bench 2.0, 89 of 267 attempts close on the verifier's acceptance, and 84% of them pass, against 57% of those that run to the time limit.
Why the judgment cannot be the worker's. Shown a counterexample, the worker takes back its own "finish" 47 times out of 49. With none in hand, it still ends after 16 of the 49 assessments that say the candidate has not been accepted.
In the revised paper: Section 4.3, lines 429–445, and Figure 6d; Section 2, lines 163–170; Appendix C.11.
Follow-up to Reviewer Gpd9
Dear Reviewer Gpd9,
Thank you again for your review. For your three questions we ran these experiments; all are in the revised paper.
- A control that isolates ProContract (Q1). ProContract switched off on the same executable, 200 tasks: +5.07 points [3.70, 6.54]; higher on 154 tasks, lower on 40 (Table 1).
- Effort and the highest-mean claim (Q2). Every run is at max effort: 89.70% mean pass rate against 87.79% for the best of 66 public entries (Section 4.2).
- Verifier feedback (Q3). Four cells on one panel of 40 tasks: keeping the task open carries the gain with no feedback at all, higher on 26 of 30 tasks (Section 4.3).
- Decoupling from the harness, which you noted. We replaced the harness under a running task: the Contract, its history and its allowance are identical before and after (Section 4.4; Appendix C.6).
Do these answer your questions? If anything remains open, or there is a further control you would like to see, please tell us; we will answer promptly.
With thanks,
The authors
Response to Reviewer Gpd9 (1/2)
Dear reviewer Gpd9,
Thank you for recognizing that the paper names two long-horizon failure modes and decouples responsibility and progress from the harness to address them, and that progress is recorded only through a careful per-acceptance rule. Your three requests are precise: the first two are answered here, the third in a second comment. Each answer stands on its own; the passages behind it are set in blue in the revised paper.
1. A control that isolates ProContract from the harness
Q1 (summary): Every baseline runs mini-SWE-agent, while ProContract is a full system that also spends more. Run ProContract's own harness with the kernel disabled, on the same runtime and tools, letting the harness mark its own tasks done, and report the 200-task distribution. TL;DR: On one executable, with the same model, the same six hours and no limit on requests in either arm, the agent that marks its own task done scores 71.22% and ProContract scores 76.28%: +5.07 points (95% interval 3.70 to 6.54), higher on 154 of the 200 tasks and lower on 40. The submission's comparison with the public run of the same model gave 5.33.
Why the submission compared with public runs. A public leaderboard run is produced by a third party, so we cannot tune it, and it holds the model fixed. Your control also holds the runtime and the tools fixed.
What we ran. We now build ProContract on Codex. "Off" is the same binary with ProContract disabled and Codex's own mode for long tasks,
/goal, enabled: the agent keeps a persistent goal, continues automatically, and marks its task done itself./goal)The cost envelope. Both arms have the same allowance: six hours and unlimited requests. The baseline spends less because it declares its task done after a median of 41 minutes. That difference is the effect we measure, not a difference in budget. Spending more does not by itself raise the score: across the 199 scored runs of ProContract with GPT-6.1 Sol, the time used is uncorrelated with the pass rate (−0.04). And where a task can be finished, ProContract spends less: on Terminal-Bench 2.0 an attempt that ends with the verifier's acceptance takes a median of 48 requests and costs $1.01 on average, against 91.5 requests and $2.43 for an attempt that works to the limit.
Between off and on. A harness that marks its own tasks done is
/goal. The systems in between, with the worker's "done" merely refused, or with the verifier kept and the worker free to end, are in our second comment.In the revised paper: Table 1; Section 4.1, lines 360–364; Section 4.3, lines 396–398 and 422–423; Appendix C.9.
2. Reasoning effort, and the highest-mean claim
Q2 (summary): The headline gain is against GPT-5.6 Sol at xhigh (69.89), but the same model at max effort scores 77.64 with mini-SWE-agent, above ProContract's 75.25. Does the highest-mean claim hold on the complete third-party population? TL;DR: The submission's comparison was matched in effort: ProContract and the public run both used xhigh (75.25% against 69.89%); the 77.64% entry is a different setting, max. In the revision every run of ours is at max effort, and against the complete public leaderboard of 66 entries ProContract with GPT-6.1 Sol has the highest mean pass rate: 89.70% against 87.79% for the best entry, at 62% of that entry's cost per task.
The cost of ProContract counts every model request of every role, the verifier included.
In the revised paper: Section 4.2, lines 371–376, and Figure 5; the complete leaderboard is Table 17 in Appendix C.9.