acceptodds
Under review as a conference paper at ICLR 2027

Interlock: From Local Correctness to Task Completion in Long-Horizon Agents

Abstract

Large language model agents increasingly undertake long-horizon tasks in complex environments characterized by fragmented evidence, partially completed states, and dependencies across resources and artifacts. Existing agents may satisfy individual requirements while failing to maintain these relationships, producing locally correct but incomplete outcomes.These failures arise because task completion depends on relationships among environmental evidence, consequential decisions, and required effects. We propose Interlock, a framework that represents each task as a typed completion graph. Grounding edges connect public evidence to decisions, coupling hyperedges impose joint constraints across components, and realization edges connect decisions to all required downstream effects. Interlock compiles this graph into a resettable workspace, a public request, and an evaluation rubric.A source-coverage review checks whether the original requirements survive generation, while a public-input review checks whether the rubric is supported by the materials available to the solver. An agent-as-judge inspects execution artifacts and runs verification scripts, retaining trajectories judged to satisfy all mandatory requirements. To evaluate training utility, we fine-tune two Qwen3.6 backbones on their own accepted trajectories. Across the two backbones, training improves AutomationBench task pass rates by 4.00–4.50 percentage points, JobBench weighted scores by 3.20–6.64 points, and Harvey LAB criterion pass rates by 3.89–4.16 percentage points. These results demonstrate the training utility of the synthesized experience across external task environments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-horizon agent, environment scaling, synthetic training data, agent post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_j7ovOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of local correctness without overall task completion in long-horizon interactive agents and proposes Interlock, a framework for task synthesis and training. The framework uses a typed completion graph to represent three types of relationships: the evidence required to support decisions, constraints that different components must jointly satisfy, and follow-up operations required after a decision. These correspond to Grounding, Coupling, and Realization, respectively. The paper uses the graph to jointly generate workspaces, task requirements, and evaluation rubrics, checks the generated tasks through source-coverage and public-input reviews, and selects trajectories using an agent-as-judge that can inspect files and run verification scripts. Finally, the authors fine-tune two Qwen3.6 backbones and evaluate the transfer of this training experience to other environments using AutomationBench, JobBench, and Harvey LAB.

Critical strengths
  1. The research problem has practical significance, and the method focuses on overall task completion. In tasks involving multiple files and constraints, a correct local decision does not imply that the corresponding updates to related records, deliverables, and states have all been completed. The paper represents evidential support, joint constraints, and downstream fulfillment separately and uses the same completion graph to constrain the generation of workspaces, task descriptions, and evaluation rubrics. This design provides a relatively clear way to organize task dependencies, check for omitted requirements, and trace the effects of decisions on subsequent outcomes. In addition, the source-coverage and public-input reviews check whether the original requirements are retained and whether evaluation criteria are supported by publicly available inputs, respectively. These checks have practical value.

  2. The complete pipeline improves performance across multiple external benchmarks, and the judge comparison is also valuable. Table 1 shows that task pass rates on AutomationBench increase by 4.00 and 4.50 percentage points for the two backbones, respectively; weighted scores on JobBench increase by 3.20–6.64 points; and criterion pass rates on Harvey LAB increase by 3.89–4.16 percentage points. These results indicate that the generated experience has training value and support its transfer across environments. In addition, Table 2 compares a text-only judge with an agent-as-judge that can inspect files and run scripts while holding the rollout pool, judge backbone, evaluation rubrics, and training budget fixed. The latter performs better on all four downstream metrics, providing relatively direct support for active verification in selecting training trajectories.

Critical weaknesses
  1. The independent benefits of the core design remain unclear. The paper presents the typed completion graph and task synthesis constrained by it as the main methodological contributions, but the current experiments do not isolate the role of this design. The comparison with ClawGym [1] differs in task content, execution length, and demonstration source, while the domain-coverage experiments and judge comparison do not directly establish the added value of the completion graph itself. I therefore acknowledge that the current results show the training utility of the complete Interlock pipeline, but it remains difficult to determine how much of the improvement comes from the proposed relational modeling and how much comes from other data or training factors. A relatively direct evaluation would start from the same original task requirements, match generation and training budgets as closely as possible, and compare the completion graph with a plain natural-language checklist of requirements, reporting both requirement retention and downstream performance. This would help clarify the core design's independent contribution.

  2. Direct evaluation of locally correct but globally incomplete behavior is still lacking. The paper emphasizes cases in which an agent makes a correct decision but does not fully execute the follow-up operations that the decision requires. However, the main experiments report overall task pass rates or aggregate scores. These metrics alone cannot distinguish improvements in upstream decision accuracy from more complete fulfillment of downstream obligations after a correct decision. The examples in the main text and appendix also show only different execution attempts by the same model, without directly comparing behavior before and after training. Appendix B defines follow-through diagnostics but does not report the corresponding results. I acknowledge the overall performance gains, but direct evidence that these gains reflect improvements in the completion behavior emphasized by the paper is still lacking. In particular, decision units for which both the pre-training and post-training models make correct upstream decisions should be compared to test whether training actually reduces omissions of required follow-up operations.

References:

[1] Fei Bai, Huatong Song, Shuang Sun, Daixuan Cheng, Yike Yang, Chuan Hao, Renyuan Li, Feng Chang, Yuan Wei, Ran Tao, et al. ClawGym: A Scalable Framework for Building Effective Claw Agents. arXiv preprint arXiv:2604.26904, 2026.

Questions for authors
  1. What independent benefits does the completion graph provide over a plain natural-language checklist of requirements? Could you directly compare the two representations starting from the same original task requirements, matching generation and training budgets as closely as possible, and report requirement retention and downstream performance? This would help distinguish benefits arising from relational modeling itself from those that may arise from other data or training factors.

  2. Does training improve the model's ability to fully fulfill the downstream obligations required by a correct decision? On tasks not used for training, could you report upstream decision accuracy, the proportion of complete downstream fulfillment conditional on a correct decision, and the proportion with both a correct upstream decision and complete downstream fulfillment, before and after training? In particular, I would like to see a comparison of decision units for which both models make correct upstream decisions, to test whether training actually reduces the targeted omissions and further clarify the relationship between overall performance gains and improvements in the specific completion behavior.

Reviewer_221eOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Interlock, a data synthesis framework for long-horizon agents. It introduces a typed completion graph with grounding, coupling, and realization relations to model dependencies among evidence, decisions, and downstream effects. The framework generates task environments, requests, and evaluation rubrics, and uses agent-as-judge to filter trajectories for rejection-sampling fine-tuning. Experiments on two Qwen backbones show improvements across multiple external benchmarks.

Critical strengths
  • The paper addresses an important problem of overall task completion in long-horizon agents, beyond the correctness of individual operations.
  • The typed completion graph provides a clear and structured way to model task dependencies and completion requirements.
  • The experiments cover multiple backbones and external benchmarks, with additional analyses of task diversity, execution length, and verification strategies.
Critical weaknesses
  • Limited baseline comparisons. The paper mainly compares against ClawGym-SFT, without evaluating other relevant task synthesis methods. Moreover, ClawGym uses released demonstrations, while Interlock uses self-generated and filtered trajectories. This makes it difficult to isolate the contribution of the proposed task synthesis method. More baselines under comparable settings would strengthen the evaluation.
  • Insufficient analysis of agent-as-judge reliability and efficiency. Section 4.4 shows improved downstream performance over LLM-as-judge but does not directly evaluate judgment accuracy or consistency. Comparisons with deterministic rule-based verifiers and repeated evaluations would be valuable. Additionally, latency and verification overhead are not reported, which are particularly important for potential online RL applications.
  • Lack of component-level ablations. Grounding, coupling, and realization are central to Interlock, but their individual contributions remain unclear. Ablating each relation type would help establish their necessity and explain the observed improvements.
Questions for authors

See Weaknesses.

2 comments

The authors' rebuttal resolves my main concerns regarding the baseline comparison, the effectiveness of the agent-as-judge, and the component analysis. Taking the other reviewers' comments into account as well, I am willing to raise my score.

Thank you very much for raising your score and for your positive assessment of our work. We appreciate your constructive feedback and would be happy to address any further questions or concerns.

Reviewer_muQ6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Interlock, a framework for synthesizing complex agent tasks while preserving dependencies among evidence, decisions, and follow-up obligations. It compiles typed completion graphs into workspaces, public-facing requests, and evaluation rubrics, separately auditing requirement preservation and the public grounding of evaluation criteria. Each backbone executes tasks repeatedly and undergoes rejection-sampling fine-tuning on its own trajectories that an agent-as-judge deems to satisfy all mandatory criteria. Using data collected from 3,278 tasks, the authors fine-tune two Qwen3.6 backbones and demonstrate improved transfer to AutomationBench, JobBench, and Harvey LAB.

Critical strengths

S1. Successful trajectories from synthesized tasks consistently improve performance across external environments. In Table 1, both backbones improve over their base models on all four evaluation metrics, with AutomationBench task success rates increasing by 4.00 and 4.50 percentage points, respectively. These results indicate that the generated experience transfers to independently authored tasks, supporting the practical value of the overall data pipeline.

Critical weaknesses

W1. The incremental value of typed completion graphs is insufficiently isolated from standard Best-of-N or judge-based selection of successful trajectories. The ClawGym comparison in Table 1 also varies the data source, contrasting public demonstrations with self-generated rollouts. Moreover, no checklist-based generation and selection baseline uses the same source requirements and generation and verification budgets. Consequently, the contribution of relational representation and tracking to external performance gains remains difficult to distinguish from the benefits of selecting high-quality trajectories for training.

W2. Direct evidence of reduced obligation omission and reliable completion judgments is limited. the authors define a measure of follow-up obligation fulfillment conditional on correct decisions but reports no results before and after training. The authors also present a case in which the judge marks all 29 criteria as satisfied despite a remaining violation of the public rules. It therefore remains unclear how effectively the proposed approach reduces the acceptance of incomplete executions as successes or whether fine-tuning reduces follow-up obligation omissions.

Questions for authors
  1. Have you conducted comparisons that construct tasks and rubrics from the same source requirements without graphs, then apply Best-of-N or success-threshold filtering while matching the backbone, candidate count, judge tool access, and training token budget? If so, please report the selection rules and actual generation and verification budgets. Since Table 2 isolates judge tool use, evidence of gains from relational representation over standard candidate selection would help address W1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.