- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces KAT, a verifier-preserving process-supervision method for reinforcement learning of coding agents. The work targets a limitation of group-relative policy optimization: when all rollouts for a task receive the same binary verifier outcome, outcome-based advantages provide no learning signal despite potentially substantial differences in trajectory quality. KAT addresses this by evaluating trajectories along several software-engineering lifecycle dimensions, including failure reproduction, root-cause diagnosis, targeted post-fix validation, regression validation, and tool-use behavior. Process signals are standardized within each verifier-outcome stratum and bounded before being combined with the outcome advantage, ensuring that process supervision refines comparisons within the same outcome class without overriding the verifier-induced ordering between successful and failed trajectories. The process evaluator further combines evidence-linked judgments from a frozen LLM with deterministic evidence validation and reliability-aware weighting. Experiments starting from Qwen3.6-35B-A3B report consistent improvements across multiple software-engineering and agentic benchmarks, together with ablations and analyses examining the proposed process signals and advantage construction.
Critical strengths
-
The paper addresses an important limitation of outcome-only reinforcement learning for coding agents. Binary executable verifiers provide reliable supervision, but they cannot distinguish trajectories when all rollouts in a group share the same outcome. The observation that substantial rollout computation may therefore provide no group-relative learning signal is well motivated and particularly relevant for long-horizon agentic tasks.
-
The verifier-preserving advantage construction is simple and principled. Rather than directly mixing outcome and process rewards with a scale-sensitive coefficient, KAT restricts process comparisons to trajectories with the same verifier outcome and bounds their contribution to the final advantage. This provides a clean separation between verifier-defined success and process-based within-outcome refinement, with a formal guarantee that process supervision cannot reverse the ordering between successful and failed trajectories.
-
The empirical results are broad and consistently positive. KAT improves over the base model across multiple benchmarks covering repository-level software engineering, terminal interaction, and scientific programming. The paper also provides extensive ablations and analyses of the advantage construction, process signals, signal coverage, and continuation behavior, which together provide useful empirical evidence for the proposed design.
Critical weaknesses
-
The performance contribution of homogeneous-group supervision is not directly isolated. Table 5 shows that KAT recovers nonzero advantages from many all-failure groups, but increased signal coverage alone does not demonstrate that training on these additional signals contributes to the final performance gains. The outcome-only comparison in Table 3 removes process supervision from both mixed-outcome and homogeneous groups simultaneously, making it difficult to attribute the improvement specifically to the latter. Similarly, the continuation analysis in Table 6 supports the predictiveness of the process signal, but does not establish the benefit of using homogeneous-group signals for policy optimization. A particularly informative ablation would retain the same KAT advantages for mixed-outcome groups while disabling updates from homogeneous groups, ideally further separating all-failure and all-success groups. Reporting the corresponding rollout costs and number of contributing groups would also help quantify the practical value of recovering these otherwise unused training signals.
-
The initial task filtering limits the scope of the homogeneous-group results. Although Table 5 demonstrates that KAT recovers nonzero advantages from all-failure groups during training, all training tasks were initially selected to have at least one successful and one failed rollout among eight samples. The results therefore establish this benefit on tasks with observed initial successes, but do not establish whether it extends to harder tasks with no successes in the initial samples. A controlled comparison that includes initially all-failure tasks, under a matched rollout budget, would clarify whether KAT can effectively exploit process supervision beyond this selected task distribution, where outcome-only methods provide no initial learning signal.
Questions for authors
-
How much of KAT's final performance improvement is specifically attributable to updates from outcome-homogeneous groups? Could the authors provide an ablation that retains the full KAT advantage construction for mixed-outcome groups but disables updates from homogeneous groups? If feasible, separately disabling all-failure and all-success groups would further clarify where the benefit comes from.
-
Does KAT remain effective when training includes tasks that are already all-failure under the initial policy? Since the current task pool retains only tasks with both successful and failed initial rollouts, an experiment including initially all-failure tasks under a matched rollout budget would help establish whether process supervision can expand the set of learnable tasks, rather than only recover additional signal after initially solvable tasks become homogeneous during on-policy training.
2 comments
Thank you for the detailed responses. The authors have addressed some of my concerns. I remain positive on this paper and keep my current rating.
Response to Reviewer EZuU
Thank you for the constructive suggestions. We have completed the proposed mask ablation, which directly tests whether updates from homogeneous groups improve downstream performance.
Homogeneous-group updates contribute to the gains. Each variant was trained once using the same training and rollout budgets. We keep the mixed-group advantage computation and pre-mask loss normalization unchanged; masked rollouts still count toward the generation budget. Evaluation uses the same 487 valid Verified tasks and 720 valid Pro tasks across all variants.
Disabling all homogeneous-group updates reduces performance by 1.85 percentage points on Verified and 1.94 points on Pro, corresponding to 9 and 14 fewer resolved tasks. Disabling only all-failure updates produces drops of 1.64 and 1.67 points, compared with 0.41 and 0.28 points when only all-success updates are disabled. Differences are calculated from the unrounded count ratios.
These results directly link recovered homogeneous-group signals to downstream performance, with all-failure supervision contributing more in these runs. The drops measure the marginal effect of each mask; they are not an additive decomposition of the total gain.
Rollout cost and contributing groups. Each run uses the same nominal budget of 122,880 rollouts (120 steps × 128 tasks × 8 rollouts). Masked rollouts still count toward this budget. A contributing group is a valid group with at least one nonzero final advantage after applying the mask.
Generalization to tasks with no initial success. Our empirical conclusion concerns the selected task pool: homogeneous-group updates improve final performance, with the larger benefit coming from all-failure groups. Initial task filtering does not remove this learning opportunity—58.4% of retained training groups are all-failure (Table 5).
Tasks with zero successes in the initial eight samples are a broader setting that we have not evaluated. The construction remains applicable whenever the combined process signal varies, but this alone does not establish performance gains on that task pool. We will clarify this scope and identify evaluation on initially all-failure tasks as future work. The new mask ablation directly supports the contribution claimed for the evaluated setting.