- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- BANC treats executable feedback for coding-agent RL as a measurement that costs resources, may be noisy, and is observed selectively. It keeps the underlying action units and branching mechanics while changing how feedback is purchased and used.
- The method combines gradient-value acquisition, verifier calibration corrected for audit selection, and augmented inverse-propensity credit estimation. Randomized purchase probabilities are logged and bounded away from zero, allowing the local credit estimator to correct for adaptive acquisition under stated assumptions.
- Experiments test the individual mechanisms before evaluating issue resolution. The main comparison uses matched token and sandbox caps, five training seeds, and a temporally held-out SWE-rebench suite, with additional benchmark, language, and model-size evaluations.
Critical strengths
- The paper identifies a real interaction between acquisition and learning: measuring high-value branch points more often changes the observed credit distribution. Logging randomized purchases makes that selection explicit. The factorial ablation is informative because acquisition alone does not help, whereas acquisition with calibration and correction performs best.
- The mechanism-level evidence goes beyond final benchmark scores. Exhaustive branching supports direct credit-recovery diagnostics; the noise experiments examine verifier calibration; and BPO+DIFF controls for the choice of credit units. Together, these tests make it easier to assess why the combined system improves.
- The end-to-end result is supported by a reasonably careful protocol. BANC improves SWE-rebench resolution from 25.6% to 28.3% over BPO+DIFF across five seeds, with a prespecified confirmatory comparison. The paper also reports weaker gains on cheap tasks and a small negative Go-transfer result, which help define where the method is useful.
Critical weaknesses
- The credit guarantee depends on calibrated reliability and conditionally independent audit abstention. Crashes, timeouts, and weak mutation coverage can depend on patch quality even after conditioning on the available features. The exhaustive recovery diagnostic retains fully determinate units, so it does not test this vulnerable part of the actual pipeline. A sensitivity analysis is needed to connect the guarantee to operational behavior.
- The comparison matches tokens and sandbox time, but BANC consumes more training compute. Appendix Table 10 reports 318.4 node GPU-hours versus 291.2 for BPO+DIFF. The overhead is disclosed, which is helpful, but the current results do not show whether the same GPU-hour allowance spent on more baseline training would close the resolution gap.
- The audit protocol is a higher-fidelity operational reference rather than an independent correctness oracle. Its properties are generated from interfaces and filtered against the reference patch; mutation coverage does not by itself establish coverage of the issue's intended behavior. The reported 12.1% disagreement therefore cannot alone determine how often the original verifier, rather than the reference protocol, is wrong. This limits the interpretation of the reliability results even if the estimator behaves as designed.
Questions for authors
- How do credit bias and issue resolution change when audit abstention remains correlated with reference correctness after conditioning on the calibrator features?
- Does BANC retain its advantage if BPO+DIFF receives the same total GPU-hour allowance, including online scoring overhead?
- For the 59 determinate raw/reference disagreements, is there an independent check of which verdict is correct, particularly for incomplete-assertion cases? This would help assess what the calibrator's improved agreement means in practice.
8 comments
Testing Informative Audit Abstention, Compute Fairness, and Reference-Verdict Quality
To keep the scenario internally consistent, we fix marginal missingness at , based on 12 abstentions among 500 uniformly audited validation leaves. As a separate parameter, we set marginal reliability to , based on agreement among the 488 determinate leaves, namely . These are different denominators and are used as fixed scenario parameters, not assertions that reliability is directly observed among naturally abstaining leaves. For each , we adjust the masked and unmasked groups' conditional reliabilities to preserve both and . The corresponding binary-model Pearson correlations and abstention odds ratios are computed from this common parameterization; credit-recovery metrics come from the controlled remasking experiments.
| MNAR severity | Binary-model Pearson | Abstention odds ratio (bad vs. good verdict) | AIPW absolute bias | RMSE | Gradient cosine similarity |
|---|---|---|---|---|---|
| 0 | 1.00 | 0.021 | 0.218 | 0.84 | |
| −0.070 | 2.74 | 0.025 | 0.224 | 0.83 | |
| −0.141 | 5.50 | 0.028 | 0.227 | 0.82 | |
| −0.235 | 12.73 | 0.031 | 0.232 | 0.81 |
As missingness becomes more informative, recovery quality degrades—as it should if the model's conditional-missingness assumptions fail. Even at the strongest tested setting (), absolute bias increases from 0.021 to 0.031, RMSE from 0.218 to 0.232, and gradient cosine similarity decreases from 0.84 to 0.81. For context, the original paper reports cosine similarity 0.51 for uncorrected selection and 0.62 for standard IPS; those numbers come from a different masking protocol, so the comparison is descriptive rather than a matched MNAR control.
We additionally evaluate a mild illustrative setting, , motivated by the background-daemon/nonzero-exit scenario in Appendix D.3. Under the selected marginal missingness rate , a two-term linear approximation for original-minus-sibling credit yields a possible incremental bias scale of . This is not a reliability gap empirically measured on natural abstentions and not a universal bound. Directly testing the natural gap would require independent adjudication of leaves for which the existing reference protocol abstains, with explicitly reported sample sizes and criteria.
2. End-to-end outcomes under alternative operational abstention rules
The controlled MNAR diagnostic above measures credit-recovery sensitivity; it is not an end-to-end training experiment under naturally induced MNAR mechanisms. We separately compared three implementable handling rules on the same three training seeds, with other settings fixed:
- Default calibrated fallback: Use calibrated soft credit when an audit abstains — 28.3% resolved rate.
- Pessimistic failure assignment: Treat a crashed, timed-out, or indeterminate audit leaf as failure — 28.1%.
- Drop the credit unit: Remove the entire unit if any of its leaves abstains — 27.6%.
The 0.2-point difference between calibrated fallback and pessimistic assignment is too small to support a strong statistical ranking by itself. Dropping abstaining units performs worse in this setting, consistent with a potential survivor-selection problem. Taken together, these results show that (i) controlled MNAR deviations impair credit recovery but leave some useful recovery in the tested range and (ii) calibrated fallback performs best among the tested online handling rules. Neither result alone identifies end-to-end performance under the unknown natural MNAR mechanism, and neither makes Equation (9) an unconditional unbiasedness guarantee.
Testing Informative Audit Abstention, Compute Fairness, and Reference-Verdict Quality
3. Equal GPU-hours: compute overhead does not eliminate the measured advantage
We agree that the original token and sandbox caps did not equalize total GPU cost. Table 10 reports 318.4 node GPU-hours for BANC, versus 291.2 for BPO+DIFF, an increase of approximately 9.3%. We will state this explicitly and provide a supplementary resource ledger (Appendix A.8) covering generated tokens, sandbox seconds, and GPU-hours for the original baseline, equal-audit-access baseline, extended-GPU baseline, and BANC.
We evaluated both directions of GPU-budget matching. Extending BPO+DIFF to 318.4 node GPU-hours raises its resolved rate from 25.6% to 25.9%, still 2.4 points below full BANC at 28.3%. Stopping BANC at 291.2 node GPU-hours yields 27.9%, still 2.3 points above the original BPO+DIFF result at that GPU allowance.
As noted in the response to Rmjz, this is a separate equal-GPU-hour sensitivity analysis. The baseline may exceed the original -token limit to use its additional compute, so it should not be mistaken for simultaneous equality of all budget dimensions. Within the tested range, simply allocating the GPU-hour difference to additional baseline training does not close the measured gap. BANC nevertheless incurs higher total training GPU cost; our efficiency claim is limited to higher resolution under the original token/sandbox caps and lower sandbox usage.
Testing Informative Audit Abstention, Compute Fairness, and Reference-Verdict Quality
We appreciate the reviewer's assessment of the selection-correction mechanism, diagnostics, and resource-controlled study. The three concerns go directly to the operational validity of BANC: whether nonrandom audit abstention distorts credit, whether additional GPU compute explains the gain, and which verifier should be trusted when the raw and reference protocols disagree. We address each using additional analyses while distinguishing empirical evidence from theoretical guarantees.
1. Informative abstention: measured prevalence and controlled MNAR stress tests
We agree that the original 512-unit exhaustive diagnostic, which retained only units with determinate reference verdicts on every leaf, cannot establish robustness to naturally informative abstention. Its purpose was to provide computable full-feedback credit for a controlled diagnostic, not to represent all abstaining units in online training.
We first measured abstention at the appropriate levels. The online protocol uses K=4: one original leaf and four siblings. In a task-stratified sample of 1,024 eligible online credit units, we restored pre-purchase states, generated missing siblings under frozen πθold, and ran the complete reference protocol on all five leaves. Seventy units had at least one abstention: 70/1024=6.84%. The unclustered Wilson 95% interval is approximately [5.45%, 8.55%]. This is an observed rate in the fully inspected diagnostic sample, not a direct count for unobserved leaves in the ordinary training logs. Unequal task-stratum inclusion probabilities would require sampling weights for extrapolation.
In the ordinary training logs, the rate at which an actually executed measurement yielded at least one abstention within a training iteration is 10.8% across five seeds (see our response to DP11). This iteration-level rate and the 6.84% diagnostic unit-level rate have different denominators and must not be interchanged. Nor does the former identify whether unpurchased branches would have abstained. We use the potential event Di(1) for that hypothetical full-branch determinacy condition.
The 1,024-unit diagnostic includes 5,120 full leaf audits; at the main experiment's 74-second mean per leaf, the reference-audit component alone is estimated at 3.79×105 sandbox seconds, before sibling generation, restoration, and execution. This is a separate diagnostic ledger, excluded from the main run's 0.080×106-second audit budget and its overall matched sandbox consumption.
In contrast, the offline exhaustive study uses K=32 sibling continuations plus one original leaf, or 33 leaves per unit. Its original candidate pool had 584 units, of which 72 contained at least one abstaining leaf; retaining the 512 fully determinate units implies a 12.3% candidate-pool unit-abstention rate. We will clarify that the abbreviated “K=32 leaves” heading in the original Table 1 meant 32 sibling continuations (33 leaves including the original).
For reference, if leaf abstention were independent with a marginal rate of 2.4%, the probability of at least one abstention would be 1−0.9765=11.44% for a five-leaf unit and 1−0.97633=55.14% for a 33-leaf unit. These are hypothetical independence benchmarks, not distribution-free bounds or evidence of a particular correlation structure. The 2.4% leaf rate was measured on a different validation population; correlation must instead be assessed from joint leaf/unit statistics on the same diagnostic population.
To probe the more consequential concern—abstention correlated with verifier correctness after conditioning on observed features—we constructed a controlled missing-not-at-random (MNAR) sensitivity experiment on the fully determinate diagnostic set. We retain the already known reference verdicts for evaluation, but impose new observation masks whose probability depends on latent raw/reference reliability. This isolates how informative missingness affects the credit estimator without pretending that naturally abstaining leaves have observable zℓref labels. It tests sensitivity to an imposed missingness mechanism, not identification of the natural one.
For a leaf, let Mℓ=1 indicate that the controlled experiment masks its outcome, and let rℓ⋆=1[yℓ=zℓref] denote reliability available for evaluation only because the original diagnostic leaf was determinate. We parameterize the conditional reliability gap as
=\Pr(r_\ell^\star=1\mid M_\ell=1,y_\ell,w_\ell) -\Pr(r_\ell^\star=1\mid M_\ell=0,y_\ell,w_\ell).$$ A negative $\Delta r$ means masked leaves tend to have less reliable raw-verifier verdicts. The actual naturally abstaining leaves do not have the requisite reference labels, so this contrast is **defined for the controlled masking analysis**, not measured directly for natural ABSTAIN events.