- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem of local correctness without overall task completion in long-horizon interactive agents and proposes Interlock, a framework for task synthesis and training. The framework uses a typed completion graph to represent three types of relationships: the evidence required to support decisions, constraints that different components must jointly satisfy, and follow-up operations required after a decision. These correspond to Grounding, Coupling, and Realization, respectively. The paper uses the graph to jointly generate workspaces, task requirements, and evaluation rubrics, checks the generated tasks through source-coverage and public-input reviews, and selects trajectories using an agent-as-judge that can inspect files and run verification scripts. Finally, the authors fine-tune two Qwen3.6 backbones and evaluate the transfer of this training experience to other environments using AutomationBench, JobBench, and Harvey LAB.
Critical strengths
-
The research problem has practical significance, and the method focuses on overall task completion. In tasks involving multiple files and constraints, a correct local decision does not imply that the corresponding updates to related records, deliverables, and states have all been completed. The paper represents evidential support, joint constraints, and downstream fulfillment separately and uses the same completion graph to constrain the generation of workspaces, task descriptions, and evaluation rubrics. This design provides a relatively clear way to organize task dependencies, check for omitted requirements, and trace the effects of decisions on subsequent outcomes. In addition, the source-coverage and public-input reviews check whether the original requirements are retained and whether evaluation criteria are supported by publicly available inputs, respectively. These checks have practical value.
-
The complete pipeline improves performance across multiple external benchmarks, and the judge comparison is also valuable. Table 1 shows that task pass rates on AutomationBench increase by 4.00 and 4.50 percentage points for the two backbones, respectively; weighted scores on JobBench increase by 3.20–6.64 points; and criterion pass rates on Harvey LAB increase by 3.89–4.16 percentage points. These results indicate that the generated experience has training value and support its transfer across environments. In addition, Table 2 compares a text-only judge with an agent-as-judge that can inspect files and run scripts while holding the rollout pool, judge backbone, evaluation rubrics, and training budget fixed. The latter performs better on all four downstream metrics, providing relatively direct support for active verification in selecting training trajectories.
Critical weaknesses
-
The independent benefits of the core design remain unclear. The paper presents the typed completion graph and task synthesis constrained by it as the main methodological contributions, but the current experiments do not isolate the role of this design. The comparison with ClawGym [1] differs in task content, execution length, and demonstration source, while the domain-coverage experiments and judge comparison do not directly establish the added value of the completion graph itself. I therefore acknowledge that the current results show the training utility of the complete Interlock pipeline, but it remains difficult to determine how much of the improvement comes from the proposed relational modeling and how much comes from other data or training factors. A relatively direct evaluation would start from the same original task requirements, match generation and training budgets as closely as possible, and compare the completion graph with a plain natural-language checklist of requirements, reporting both requirement retention and downstream performance. This would help clarify the core design's independent contribution.
-
Direct evaluation of locally correct but globally incomplete behavior is still lacking. The paper emphasizes cases in which an agent makes a correct decision but does not fully execute the follow-up operations that the decision requires. However, the main experiments report overall task pass rates or aggregate scores. These metrics alone cannot distinguish improvements in upstream decision accuracy from more complete fulfillment of downstream obligations after a correct decision. The examples in the main text and appendix also show only different execution attempts by the same model, without directly comparing behavior before and after training. Appendix B defines follow-through diagnostics but does not report the corresponding results. I acknowledge the overall performance gains, but direct evidence that these gains reflect improvements in the completion behavior emphasized by the paper is still lacking. In particular, decision units for which both the pre-training and post-training models make correct upstream decisions should be compared to test whether training actually reduces omissions of required follow-up operations.
References:
[1] Fei Bai, Huatong Song, Shuang Sun, Daixuan Cheng, Yike Yang, Chuan Hao, Renyuan Li, Feng Chang, Yuan Wei, Ran Tao, et al. ClawGym: A Scalable Framework for Building Effective Claw Agents. arXiv preprint arXiv:2604.26904, 2026.
Questions for authors
-
What independent benefits does the completion graph provide over a plain natural-language checklist of requirements? Could you directly compare the two representations starting from the same original task requirements, matching generation and training budgets as closely as possible, and report requirement retention and downstream performance? This would help distinguish benefits arising from relational modeling itself from those that may arise from other data or training factors.
-
Does training improve the model's ability to fully fulfill the downstream obligations required by a correct decision? On tasks not used for training, could you report upstream decision accuracy, the proportion of complete downstream fulfillment conditional on a correct decision, and the proportion with both a correct upstream decision and complete downstream fulfillment, before and after training? In particular, I would like to see a comparison of decision units for which both models make correct upstream decisions, to test whether training actually reduces the targeted omissions and further clarify the relationship between overall performance gains and improvements in the specific completion behavior.
The authors' rebuttal resolves my main concerns regarding the baseline comparison, the effectiveness of the agent-as-judge, and the component analysis. Taking the other reviewers' comments into account as well, I am willing to raise my score.