acceptodds
Under review as a conference paper at ICLR 2027

When the Verifier Is Silent: Process Advantages for Agentic Coding

Abstract

Reinforcement learning for agentic software engineering benefits from strict executable verifiers, but their binary feedback is sparse. In group-relative policy optimization, a task provides no outcome-based learning signal when all sampled rollouts receive the same verifier outcome, despite substantial differences in their process quality. Adding dense process rewards creates a structural tension: small coefficients may vanish under outcome normalization, whereas large coefficients can overturn the verifier-induced success or failure ordering. We introduce KAT, a verifier-preserving process-supervision method that makes process signals subordinate by construction. KAT partitions rollouts by executable outcome, standardizes process signals within each outcome stratum, and combines bounded process advantages with margin-scaled outcome advantages. This guarantees that no judge output or nonnegative process weight can rank an unverified trajectory above a verified one, while outcome-homogeneous groups still provide learning signals whenever process quality varies. KAT derives process signals from a frozen software-engineering lifecycle rubric covering reproduction, root-cause diagnosis, post-fix and regression validation, and tool use. An LLM judge performs evidence-linked structured matching, while deterministic components validate evidence and aggregate scores using reliability-based weights. Starting from Qwen3.6-35B-A3B, KAT improves pass@1 from 64.40% to 69.40% on SWE-bench Verified, 57.00% to 63.00% on SWE-bench Multilingual, 40.63% to 45.96% on SWE-bench Pro, and 32.02% to 41.02% on Terminal-Bench 2.1. It also transfers to SciCode, improving pass@1 from 37.53% to 44.20%. Controlled experiments show that KAT provides usable supervision and outperforms outcome-only and additive-reward alternatives. Models will be publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agentic coding, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EZuUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces KAT, a verifier-preserving process-supervision method for reinforcement learning of coding agents. The work targets a limitation of group-relative policy optimization: when all rollouts for a task receive the same binary verifier outcome, outcome-based advantages provide no learning signal despite potentially substantial differences in trajectory quality. KAT addresses this by evaluating trajectories along several software-engineering lifecycle dimensions, including failure reproduction, root-cause diagnosis, targeted post-fix validation, regression validation, and tool-use behavior. Process signals are standardized within each verifier-outcome stratum and bounded before being combined with the outcome advantage, ensuring that process supervision refines comparisons within the same outcome class without overriding the verifier-induced ordering between successful and failed trajectories. The process evaluator further combines evidence-linked judgments from a frozen LLM with deterministic evidence validation and reliability-aware weighting. Experiments starting from Qwen3.6-35B-A3B report consistent improvements across multiple software-engineering and agentic benchmarks, together with ablations and analyses examining the proposed process signals and advantage construction.

Critical strengths
  1. The paper addresses an important limitation of outcome-only reinforcement learning for coding agents. Binary executable verifiers provide reliable supervision, but they cannot distinguish trajectories when all rollouts in a group share the same outcome. The observation that substantial rollout computation may therefore provide no group-relative learning signal is well motivated and particularly relevant for long-horizon agentic tasks.

  2. The verifier-preserving advantage construction is simple and principled. Rather than directly mixing outcome and process rewards with a scale-sensitive coefficient, KAT restricts process comparisons to trajectories with the same verifier outcome and bounds their contribution to the final advantage. This provides a clean separation between verifier-defined success and process-based within-outcome refinement, with a formal guarantee that process supervision cannot reverse the ordering between successful and failed trajectories.

  3. The empirical results are broad and consistently positive. KAT improves over the base model across multiple benchmarks covering repository-level software engineering, terminal interaction, and scientific programming. The paper also provides extensive ablations and analyses of the advantage construction, process signals, signal coverage, and continuation behavior, which together provide useful empirical evidence for the proposed design.

Critical weaknesses
  1. The performance contribution of homogeneous-group supervision is not directly isolated. Table 5 shows that KAT recovers nonzero advantages from many all-failure groups, but increased signal coverage alone does not demonstrate that training on these additional signals contributes to the final performance gains. The outcome-only comparison in Table 3 removes process supervision from both mixed-outcome and homogeneous groups simultaneously, making it difficult to attribute the improvement specifically to the latter. Similarly, the continuation analysis in Table 6 supports the predictiveness of the process signal, but does not establish the benefit of using homogeneous-group signals for policy optimization. A particularly informative ablation would retain the same KAT advantages for mixed-outcome groups while disabling updates from homogeneous groups, ideally further separating all-failure and all-success groups. Reporting the corresponding rollout costs and number of contributing groups would also help quantify the practical value of recovering these otherwise unused training signals.

  2. The initial task filtering limits the scope of the homogeneous-group results. Although Table 5 demonstrates that KAT recovers nonzero advantages from all-failure groups during training, all training tasks were initially selected to have at least one successful and one failed rollout among eight samples. The results therefore establish this benefit on tasks with observed initial successes, but do not establish whether it extends to harder tasks with no successes in the initial samples. A controlled comparison that includes initially all-failure tasks, under a matched rollout budget, would clarify whether KAT can effectively exploit process supervision beyond this selected task distribution, where outcome-only methods provide no initial learning signal.

Questions for authors
  1. How much of KAT's final performance improvement is specifically attributable to updates from outcome-homogeneous groups? Could the authors provide an ablation that retains the full KAT advantage construction for mixed-outcome groups but disables updates from homogeneous groups? If feasible, separately disabling all-failure and all-success groups would further clarify where the benefit comes from.

  2. Does KAT remain effective when training includes tasks that are already all-failure under the initial policy? Since the current task pool retains only tasks with both successful and failed initial rollouts, an experiment including initially all-failure tasks under a matched rollout budget would help establish whether process supervision can expand the set of learnable tasks, rather than only recover additional signal after initially solvable tasks become homogeneous during on-policy training.

2 comments
AuthorsOpenReview ↗

Response to Reviewer EZuU

Thank you for the constructive suggestions. We have completed the proposed mask ablation, which directly tests whether updates from homogeneous groups improve downstream performance.

Homogeneous-group updates contribute to the gains. Each variant was trained once using the same training and rollout budgets. We keep the mixed-group advantage computation and pre-mask loss normalization unchanged; masked rollouts still count toward the generation budget. Evaluation uses the same 487 valid Verified tasks and 720 valid Pro tasks across all variants.

MethodVerified: C/487Pro: C/720
Full KAT338 (69.40%)331 (45.97%)
All homogeneous updates off329 (67.56%)317 (44.03%)
All-failure updates off330 (67.76%)319 (44.31%)
All-success updates off336 (68.99%)329 (45.69%)

Disabling all homogeneous-group updates reduces performance by 1.85 percentage points on Verified and 1.94 points on Pro, corresponding to 9 and 14 fewer resolved tasks. Disabling only all-failure updates produces drops of 1.64 and 1.67 points, compared with 0.41 and 0.28 points when only all-success updates are disabled. Differences are calculated from the unrounded count ratios.

These results directly link recovered homogeneous-group signals to downstream performance, with all-failure supervision contributing more in these runs. The drops measure the marginal effect of each mask; they are not an additive decomposition of the total gain.

Rollout cost and contributing groups. Each run uses the same nominal budget of 122,880 rollouts (120 steps × 128 tasks × 8 rollouts). Masked rollouts still count toward this budget. A contributing group is a valid group with at least one nonzero final advantage after applying the mask.

MethodNominal rollouts per runContributing groups
Full KAT120,00011,468
All homogeneous updates off120,0005,279
All-failure updates off120,0005,743
All-success updates off120,00010,625

Generalization to tasks with no initial success. Our empirical conclusion concerns the selected task pool: homogeneous-group updates improve final performance, with the larger benefit coming from all-failure groups. Initial task filtering does not remove this learning opportunity—58.4% of retained training groups are all-failure (Table 5).

Tasks with zero successes in the initial eight samples are a broader setting that we have not evaluated. The construction remains applicable whenever the combined process signal varies, but this alone does not establish performance gains on that task pool. We will clarify this scope and identify evaluation on initially all-failure tasks as future work. The new mask ablation directly supports the contribution claimed for the evaluated setting.

Thank you for the detailed responses. The authors have addressed some of my concerns. I remain positive on this paper and keep my current rating.

Reviewer_w4HgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem that outcome-homogeneous groups in group-relative RL cannot provide a binary reward advantage, and proposes KAT. The method separately normalizes software engineering process signals within the success and failure layers, bounds the synthesized process advantage with tanh, and then adds it to outcome advantages with sufficient margin. It preserves the advantage ordering in which successful trajectories within the same group rank above failed trajectories, and restores supervision for homogeneous groups when process quality varies. Another component is a lifecycle rubric that binds task scenarios, cites trajectory evidence, and is calibrated by reliability.

Critical strengths
  1. The homogeneous groups of results waste expensive rollout budgets, which is a practical problem in agent-based reinforcement learning. KAT intervenes at the advantage construction stage, does not require training a critic, and clearly distinguishes between result priorities and intra-layer process comparisons.
  2. The bounded process terms and result intervals in Equations (8) to (10) provide clear within-group order-preserving properties.
  3. The scenario binding, evidence temporal sequence check, distinction between environment blocking and agent errors, and dimensional reliability calibration, are more auditable than directly using a free-form LLM score.
Critical weaknesses
  1. Section 4.3 defines “no stratification” as replacing within-stratum normalization with pooled normalization. If tanh and Mout≥1M_{out}\geq1 are retained, the cross-outcome ordering proof still applies, which conflicts with the 1.9% reversal rate in Table 3.
  2. Moreover, Eq. (4) and Table 1 imply Rrub∈[−1,1]R^{rub}\in[-1,1] and η=0.3\eta=0.3, giving a minimum success–failure additive-reward gap of 0.4. Group standardization preserves that ordering, so the reported 1.4% reversal also needs explanation.
  3. The paper does not provide the complete rubric, state-to-utility mapping, or concrete binder/judge configuration.
Questions for authors
  1. Please provide the actual advantage computation for the pooled-normalization and additive-reward variants, the selected η\eta, and the precise quantity/pairs used for reversal measurement.
  2. How can 1.9% arise if bounding and the outcome margin are retained?

This work has solid merit, but there exists an inconsistency between the key ablation studies and the evaluation protocol. I will consider raising my score if the authors are able to deliver a consistent, properly aligned implementation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer w4Hg

Thank you for the precise analysis and for recognizing the method's strengths. You are correct: under the stated settings, both the pooled-normalization and additive variants preserve within-group success–failure ordering. We apologize for the mismatch between the reported diagnostic and this property.

Advantage definitions. For pooled normalization, we replace within-stratum standardization with standardization over the entire retained rollout group, keeping the other components unchanged:

=1.5 A_i^{\mathrm{out}}+ \tanh\!\left(0.3A_i^{\mathrm{ver,pool}}+ 0.5A_i^{\mathrm{rub,pool}}+0.2A_i^{\mathrm{beh,pool}}\right).$$ Each process signal uses the whole group's mean and sample standard deviation, with ε = 10⁻⁶. A signal is set to zero for singleton groups or when its standard deviation is at most 0.05. Bounding and the outcome margin remain intact, so this variant cannot produce within-group reversals. For the additive variant, the selected coefficient is **η = 0.3**: $$r_i=y_i+0.3\left(S_i^{\mathrm{life}}-P_i^{\mathrm{beh}}\right), \qquad \widehat A_i^{\mathrm{add}} =\frac{r_i-\overline r_{\mathcal G}} {s_{\mathcal G}(r)+10^{-6}}.$$ Here, the rubric scale and penalty cap are both 1. The raw success–failure gap is therefore at least 0.4, and common group standardization preserves its sign. **Why Table 3 reports nonzero reversals.** The diagnostic combined advantages from different rollout groups before forming success–failure pairs. Because normalization uses each group's own reference statistics, these values are not calibrated for cross-group ranking. The reported 1.9% and 1.4% therefore do not measure violations of the within-group property and should not have been presented as such. We will recompute the diagnostic using final trajectory advantages before importance-ratio weighting or clipping, pairing only trajectories from the same task, rollout group, and training step: $$\mathrm{Rev}=100\, \frac{\sum_g\sum_{i\in g:y_i=1}\sum_{j\in g:y_j=0} \mathbf{1}[\widehat A_i\leq\widehat A_j]} {\sum_g n_g^+n_g^-}.$$ We will apply this definition consistently to every variant, including Full KAT. Stratification's role is to define meaningful process comparisons within each outcome class; bounding and the margin separately protect verifier priority. Neither contribution requires these two baselines to reverse ordering at the chosen coefficients. **Rubric and state mapping.** The following summarizes the explicit reference specification. Lifecycle criteria are ordered prerequisites; behavioral criteria identify observed violations. | Dimension | Criteria | | --- | --- | | Reproduction | (1) Complete a relevant pre-edit check; (2) observe the reported failure; (3) cover all bound scenarios and constraints. | | Diagnosis | (1) Locate the relevant implementation; (2) explain the causal mechanism; (3) support it with additional inspection or execution addressing a plausible competing cause. | | Targeted validation | (1) Complete a relevant post-edit check; (2) demonstrate the expected behavior for at least one scenario; (3) cover all scenarios and constraints with passing evidence. | | Regression validation | (1) Complete a post-edit check beyond the issue scenarios; (2) obtain passing evidence; (3) cover a neighboring behavior or boundary at risk from the change. | | Behavioral violations | Avoidable invalid invocation; repeated agent-caused invocation error without adaptation; redundant execution without diagnostic purpose; a success claim contradicted by unresolved execution evidence; weakening verification to obtain apparent success. | For each lifecycle dimension, utilities are **0, 0.25, 0.70, and 1.00** when the first criterion is unmet, only the first is satisfied, the first two are satisfied, or all three are satisfied. Inapplicable dimensions are masked. A corroborated environment-blocked validation or reproduction attempt receives 0.25, preserving any higher earned utility; static diagnosis remains independently scoreable. Legitimate diagnostic reruns and issue-requested test maintenance are exempt from behavioral penalties. The binder extracts issue-grounded scenarios once per task. The judge emits Boolean verdicts with event IDs and quotations; deterministic checks enforce citation validity, timing relative to the first non-test source edit, and execution metadata. Prior lifecycle weights are (1, 1.5, 1.25, 0.75), calibrated by w × max(0, (q − 0.8)/0.2), where q = (correct + 1)/(total + 2). We will make the full rubric and judging prompts available, together with the model versions and decoding configuration.

Thank you for the author's painstaking clarification and response. However, given the current state of the manuscript and rebuttal, I prefer to maintain my evaluation.

Reviewer_EYVXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adds process feedback to coding-agent training while ensuring successful runs remain ranked above failures. This addresses a useful problem, groups of uniformly succ. or failed attempts otherwise provide no outcome learning signal.

Critical strengths

The proposed idea is timely and useful. The bounded advantage construction is clear, and grounding judgments in task-specific evidence is a sensible design. The continuation experiment also provides some support for the usefulness of the process scores.

Critical weaknesses

My main concern is several experiments are inconsistent with the stated setup. The no stratification variant changes normalization but retains the bounded process term, so the success–failure ordering guarantee should still hold. Table 3 nevertheless reports 1.9% reversals. The additive baseline’s reported 1.4% reversals also appear incompatible with its stated reward range and coefficient. These are central comparisons, so understanding exactly what was implemented matters for assessing the contribution. The evaluation description also specifies one sample on each of 500 SWE-bench Verified tasks, which should produce scores in increments of 0.2 percentage points. Several ablation scores, including 67.08%, do not fit this description.

Questions for authors

Could the authors clarify the actual denominators, averaging procedure, and parameters used in Table 3? Providing the full rubric and judging prompts would also help make the method reproducible.

2 comments
AuthorsOpenReview ↗

Response to Reviewer EYVX (Part 1/2)

Thank you for the careful review. We rechecked the evaluation logs and confirmed that sandbox-related failures affected the valid-evaluation counts. We apologize for not making the denominators and diagnostic scope clear in the original manuscript.

Evaluation counts and averaging. Each configuration was trained once and evaluated with one sampled attempt per task, without averaging across seeds or repeated runs. Infrastructure-invalid evaluations were excluded; model-attributable failures remained included. The additive baseline's 67.08% on SWE-bench Verified corresponds to 328/489, with 11 evaluations excluded. The outcome-only result of 41.35% on SWE-bench Pro corresponds to 301/728, with 3 evaluations excluded. We will report resolved/valid counts and exclusion reasons for each entry. “Avg.” is the unweighted mean of the two benchmark percentages, not an average over repeated runs.

Reversal rates. You are correct that neither variant should exhibit within-group reversals under the stated settings. The discrepancy comes from the diagnostic's pairing step: advantages were computed separately for each rollout group, but success–failure pairs were then formed across groups. Because normalization uses each group's own reference statistics, these values express relative standing within a group, not a common ranking across groups. A success in one group can therefore rank below a failure in another without violating either group's ordering.

The reported 1.9% and 1.4% consequently measure cross-group comparisons, not violations of the within-group property. We apologize for presenting them as such. With bounding and the outcome margin retained, the pooled variant remains order-preserving; at η = 0.3, the additive variant also retains a minimum raw reward gap of 0.4. We will recompute the diagnostic using only pairs within the same task, rollout group, and training step, and update Table 3 consistently for all variants, including Full KAT.

Stratification's role is to compare process quality among trajectories with the same outcome. KAT's bound and margin separately guarantee verifier priority without requiring a small process coefficient. These properties do not depend on the two baselines exhibiting reversals at the chosen settings.

Parameters. The specified settings are an outcome margin of 1.5, process coefficients of (0.3, 0.5, 0.2) for verifier progress, lifecycle quality, and behavioral quality, respectively, F2P/P2P weights of (0.7, 0.3), a standard-deviation threshold of 0.05, and ε = 10⁻⁶. The additive coefficient is 0.3, with rubric scale and penalty cap both set to 1.

AuthorsOpenReview ↗

Response to Reviewer EYVX (Part 2/2)

Criterion-level rubric. We give the explicit reference specification here. The binder extracts scenarios and applicability from the issue description alone, shared across all rollouts for that task. Each scenario records the trigger, observed behavior, expected behavior, and stated constraints, grounded in issue quotations. The judge returns a Boolean verdict and supporting event IDs and quotations for each criterion; it does not assign numerical rewards.

Prefixes: rep = reproduction; diag = diagnosis; tval = targeted validation; reg = regression validation; beh = behavioral violations.

CriterionRequired evidence
rep.executeA task-relevant executable check completed before the first non-test source edit.
rep.matchThe observed failure matches a bound issue scenario, rather than a setup or invocation error.
rep.coveragePre-edit failure evidence covers every bound scenario and its stated constraints.
diag.localizeInspection identifies a specific implementation location relevant to the issue.
diag.causalAn explanation connects the inspected code to the observed and expected behavior.
diag.checkAn additional inspection or execution supports the mechanism and addresses a plausible competing cause.
tval.executeA task-relevant executable check completed after the first non-test source edit.
tval.passPassing execution evidence establishes the expected behavior of at least one bound scenario.
tval.coveragePassing evidence covers all bound scenarios and their stated constraints.
reg.executeA completed post-edit check exercises behavior beyond the bound issue scenarios.
reg.passAt least one such beyond-scenario check passes.
reg.coveragePassing checks cover a neighboring behavior or boundary at risk from the change.
beh.invalid_invocationAn avoidable invocation violates a known tool contract.
beh.repeated_errorThe same agent-caused invocation error recurs without adapting the command or repository state.
beh.redundant_executionAn identical completed invocation is repeated on the same revision without a diagnostic purpose.
beh.unsupported_successAn explicit success claim contradicts a cited execution failure that remains unresolved at the time of the claim.
beh.test_bypassAn edit weakens verification to obtain apparent success instead of implementing the requested behavior.

State-to-utility mapping. For each lifecycle dimension, the three criteria above are ordered prerequisites. The highest satisfied prefix determines the utility:

UtilityState and requirement
0No supported progress — First criterion not satisfied.
0.25Partial — First criterion only.
0.70Supported — First two criteria.
1.00Complete — All three criteria.

Inapplicable dimensions are masked, not scored as zero. For reproduction and validation, a relevant attempted check blocked by independently corroborated infrastructure failure receives attempt-level utility of 0.25, preserving any higher earned utility, with no extra omission penalty. Unattempted checks and agent-caused invocation errors receive zero unless stronger evidence was already earned. Diagnosis can receive credit from static evidence even when reproduction is blocked. Legitimate failing experiments, deliberate flakiness checks, and issue-requested test maintenance are not behavioral violations.

Weights and evidence checks. Prior lifecycle weights are (1, 1.5, 1.25, 0.75); behavioral weights, in the order listed above, are (0.1, 0.2, 0.1, 0.2, 0.3). For each component, audit correctness counts give q = (correct + 1)/(total + 2), and the calibrated weight is w × max(0, (q − 0.8)/0.2). Uncalibrated components have zero weight. Lifecycle utilities are averaged over applicable dimensions using these weights and ε in the denominator; accepted behavioral penalties are summed and capped at 1. The scalar rubric reward is the lifecycle score minus the behavioral penalty.

The deterministic checker verifies citation existence, quotations, event order, and execution metadata. Reproduction must precede the first non-test source edit; validation must follow it; regression requires evidence beyond the issue scenarios. Infrastructure blockers require independent execution records. Semantic relevance remains the judge's responsibility and is assessed through the human audit; citation validity alone does not establish correctness.

We will make the full rubric and judging prompts available.

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their rebuttals. As the author-reviewer discussion period is coming to a close soon, please read their responses and engage in the discussion. Thank you for your time and effort.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.