acceptodds
Under review as a conference paper at ICLR 2027

Executable Feedback Is a Measurement: Budgeted Acquisition and Selection-Corrected Credit for Coding Agents

Abstract

Reinforcement learning for coding agents depends on executable feedback that is costly to obtain and unreliable once obtained. Branching methods reduce cost by observing counterfactual continuations only at chosen states, but the measurements they buy are then a selected sample of the eligible credit units, and the verifier at each leaf can be wrong: on a uniform audit of validation leaves, the raw verifier disagrees with a higher-fidelity reference on 12.1% of leaves with determinate reference verdicts. We introduce BANC, a training-time layer that treats executable feedback as a budgeted, randomized measurement process. BANC purchases branch and audit measurements by their expected effect on the policy gradient against shadow-priced resource costs, calibrates raw verdicts with a selection-corrected reliability model, and recovers action-level credit with an augmented inverse-propensity estimator. Because every purchase probability is randomized, bounded below, and logged, adaptive acquisition changes which measurements are bought but not the credit estimand. Tested separately, the three mechanisms cut normalized gradient error from 0.50 to 0.34, reduce harmful update mass under 20% verifier noise from 24.7% to 9.1%, and reduce the selection bias of adaptive acquisition from 0.148 to 0.021 at 44% lower RMSE than inverse propensity scoring. Under matched token and sandbox caps, BANC improves resolved rate on a temporally held-out SWE-rebench suite by 2.7 points over BPO+Diff (p = 0.019) while using 7.4% less sandbox time than APPO, with gains concentrated on long, costly tasks and preserved on 16B and 32B backbones.

Primary area:
reinforcement learning
Keywords:
Coding Agents, Reinforcement Learning, Credit Assignment, Adaptive Acquisition, Verifier Reliability, Selection Bias Correction

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_jNKQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary
  1. BANC treats executable feedback for coding-agent RL as a measurement that costs resources, may be noisy, and is observed selectively. It keeps the underlying action units and branching mechanics while changing how feedback is purchased and used.
  2. The method combines gradient-value acquisition, verifier calibration corrected for audit selection, and augmented inverse-propensity credit estimation. Randomized purchase probabilities are logged and bounded away from zero, allowing the local credit estimator to correct for adaptive acquisition under stated assumptions.
  3. Experiments test the individual mechanisms before evaluating issue resolution. The main comparison uses matched token and sandbox caps, five training seeds, and a temporally held-out SWE-rebench suite, with additional benchmark, language, and model-size evaluations.
Critical strengths
  1. The paper identifies a real interaction between acquisition and learning: measuring high-value branch points more often changes the observed credit distribution. Logging randomized purchases makes that selection explicit. The factorial ablation is informative because acquisition alone does not help, whereas acquisition with calibration and correction performs best.
  2. The mechanism-level evidence goes beyond final benchmark scores. Exhaustive branching supports direct credit-recovery diagnostics; the noise experiments examine verifier calibration; and BPO+DIFF controls for the choice of credit units. Together, these tests make it easier to assess why the combined system improves.
  3. The end-to-end result is supported by a reasonably careful protocol. BANC improves SWE-rebench resolution from 25.6% to 28.3% over BPO+DIFF across five seeds, with a prespecified confirmatory comparison. The paper also reports weaker gains on cheap tasks and a small negative Go-transfer result, which help define where the method is useful.
Critical weaknesses
  1. The credit guarantee depends on calibrated reliability and conditionally independent audit abstention. Crashes, timeouts, and weak mutation coverage can depend on patch quality even after conditioning on the available features. The exhaustive recovery diagnostic retains fully determinate units, so it does not test this vulnerable part of the actual pipeline. A sensitivity analysis is needed to connect the guarantee to operational behavior.
  2. The comparison matches tokens and sandbox time, but BANC consumes more training compute. Appendix Table 10 reports 318.4 node GPU-hours versus 291.2 for BPO+DIFF. The overhead is disclosed, which is helpful, but the current results do not show whether the same GPU-hour allowance spent on more baseline training would close the resolution gap.
  3. The audit protocol is a higher-fidelity operational reference rather than an independent correctness oracle. Its properties are generated from interfaces and filtered against the reference patch; mutation coverage does not by itself establish coverage of the issue's intended behavior. The reported 12.1% disagreement therefore cannot alone determine how often the original verifier, rather than the reference protocol, is wrong. This limits the interpretation of the reliability results even if the estimator behaves as designed.
Questions for authors
  1. How do credit bias and issue resolution change when audit abstention remains correlated with reference correctness after conditioning on the calibrator features?
  2. Does BANC retain its advantage if BPO+DIFF receives the same total GPU-hour allowance, including online scoring overhead?
  3. For the 59 determinate raw/reference disagreements, is there an independent check of which verdict is correct, particularly for incomplete-assertion cases? This would help assess what the calibrator's improved agreement means in practice.
8 comments
AuthorsOpenReview ↗

Testing Informative Audit Abstention, Compute Fairness, and Reference-Verdict Quality

We appreciate the reviewer's assessment of the selection-correction mechanism, diagnostics, and resource-controlled study. The three concerns go directly to the operational validity of BANC: whether nonrandom audit abstention distorts credit, whether additional GPU compute explains the gain, and which verifier should be trusted when the raw and reference protocols disagree. We address each using additional analyses while distinguishing empirical evidence from theoretical guarantees.

1. Informative abstention: measured prevalence and controlled MNAR stress tests

We agree that the original 512-unit exhaustive diagnostic, which retained only units with determinate reference verdicts on every leaf, cannot establish robustness to naturally informative abstention. Its purpose was to provide computable full-feedback credit for a controlled diagnostic, not to represent all abstaining units in online training.

We first measured abstention at the appropriate levels. The online protocol uses K=4K=4: one original leaf and four siblings. In a task-stratified sample of 1,024 eligible online credit units, we restored pre-purchase states, generated missing siblings under frozen πθold\pi_{\theta_{\mathrm{old}}}, and ran the complete reference protocol on all five leaves. Seventy units had at least one abstention: 70/1024=6.84%70/1024=6.84\%. The unclustered Wilson 95% interval is approximately [5.45%, 8.55%]. This is an observed rate in the fully inspected diagnostic sample, not a direct count for unobserved leaves in the ordinary training logs. Unequal task-stratum inclusion probabilities would require sampling weights for extrapolation.

In the ordinary training logs, the rate at which an actually executed measurement yielded at least one abstention within a training iteration is 10.8% across five seeds (see our response to DP11). This iteration-level rate and the 6.84% diagnostic unit-level rate have different denominators and must not be interchanged. Nor does the former identify whether unpurchased branches would have abstained. We use the potential event Di(1)D_i^{(1)} for that hypothetical full-branch determinacy condition.

The 1,024-unit diagnostic includes 5,120 full leaf audits; at the main experiment's 74-second mean per leaf, the reference-audit component alone is estimated at 3.79×1053.79\times10^5 sandbox seconds, before sibling generation, restoration, and execution. This is a separate diagnostic ledger, excluded from the main run's 0.080×1060.080\times10^6-second audit budget and its overall matched sandbox consumption.

In contrast, the offline exhaustive study uses K=32K=32 sibling continuations plus one original leaf, or 33 leaves per unit. Its original candidate pool had 584 units, of which 72 contained at least one abstaining leaf; retaining the 512 fully determinate units implies a 12.3% candidate-pool unit-abstention rate. We will clarify that the abbreviated “K=32K=32 leaves” heading in the original Table 1 meant 32 sibling continuations (33 leaves including the original).

For reference, if leaf abstention were independent with a marginal rate of 2.4%, the probability of at least one abstention would be 1−0.9765=11.44%1-0.976^5=11.44\% for a five-leaf unit and 1−0.97633=55.14%1-0.976^{33}=55.14\% for a 33-leaf unit. These are hypothetical independence benchmarks, not distribution-free bounds or evidence of a particular correlation structure. The 2.4% leaf rate was measured on a different validation population; correlation must instead be assessed from joint leaf/unit statistics on the same diagnostic population.

To probe the more consequential concern—abstention correlated with verifier correctness after conditioning on observed features—we constructed a controlled missing-not-at-random (MNAR) sensitivity experiment on the fully determinate diagnostic set. We retain the already known reference verdicts for evaluation, but impose new observation masks whose probability depends on latent raw/reference reliability. This isolates how informative missingness affects the credit estimator without pretending that naturally abstaining leaves have observable zℓrefz_\ell^{ref} labels. It tests sensitivity to an imposed missingness mechanism, not identification of the natural one.

For a leaf, let Mℓ=1M_\ell=1 indicate that the controlled experiment masks its outcome, and let rℓ⋆=1[yℓ=zℓref]r_\ell^\star=\mathbf1[y_\ell=z_\ell^{ref}] denote reliability available for evaluation only because the original diagnostic leaf was determinate. We parameterize the conditional reliability gap as

=\Pr(r_\ell^\star=1\mid M_\ell=1,y_\ell,w_\ell) -\Pr(r_\ell^\star=1\mid M_\ell=0,y_\ell,w_\ell).$$ A negative $\Delta r$ means masked leaves tend to have less reliable raw-verifier verdicts. The actual naturally abstaining leaves do not have the requisite reference labels, so this contrast is **defined for the controlled masking analysis**, not measured directly for natural ABSTAIN events.
AuthorsOpenReview ↗

Testing Informative Audit Abstention, Compute Fairness, and Reference-Verdict Quality

To keep the scenario internally consistent, we fix marginal missingness at πmiss=0.024\pi_{\mathrm{miss}}=0.024, based on 12 abstentions among 500 uniformly audited validation leaves. As a separate parameter, we set marginal reliability to rˉ=0.879\bar r=0.879, based on agreement among the 488 determinate leaves, namely (488−59)/488≈87.9%(488-59)/488\approx87.9\%. These are different denominators and are used as fixed scenario parameters, not assertions that reliability is directly observed among naturally abstaining leaves. For each Δr∈{0,−0.15,−0.30,−0.50}\Delta r\in\{0,-0.15,-0.30,-0.50\}, we adjust the masked and unmasked groups' conditional reliabilities to preserve both rˉ=0.879\bar r=0.879 and πmiss=0.024\pi_{\mathrm{miss}}=0.024. The corresponding binary-model Pearson correlations and abstention odds ratios are computed from this common parameterization; credit-recovery metrics come from the controlled remasking experiments.

MNAR severityBinary-model Pearson ρ\rhoAbstention odds ratio (bad vs. good verdict)AIPW absolute biasRMSEGradient cosine similarity
Δr=0\Delta r=001.000.0210.2180.84
Δr=−0.15\Delta r=-0.15−0.0702.740.0250.2240.83
Δr=−0.30\Delta r=-0.30−0.1415.500.0280.2270.82
Δr=−0.50\Delta r=-0.50−0.23512.730.0310.2320.81

As missingness becomes more informative, recovery quality degrades—as it should if the model's conditional-missingness assumptions fail. Even at the strongest tested setting (Δr=−0.50\Delta r=-0.50), absolute bias increases from 0.021 to 0.031, RMSE from 0.218 to 0.232, and gradient cosine similarity decreases from 0.84 to 0.81. For context, the original paper reports cosine similarity 0.51 for uncorrected selection and 0.62 for standard IPS; those numbers come from a different masking protocol, so the comparison is descriptive rather than a matched MNAR control.

We additionally evaluate a mild illustrative setting, Δr=−0.038\Delta r=-0.038, motivated by the background-daemon/nonzero-exit scenario in Appendix D.3. Under the selected marginal missingness rate πmiss=0.024\pi_{\mathrm{miss}}=0.024, a two-term linear approximation for original-minus-sibling credit yields a possible incremental bias scale of 2πmiss∣Δr∣≈0.00182\pi_{\mathrm{miss}}|\Delta r|\approx0.0018. This is not a reliability gap empirically measured on natural abstentions and not a universal bound. Directly testing the natural gap would require independent adjudication of leaves for which the existing reference protocol abstains, with explicitly reported sample sizes and criteria.

2. End-to-end outcomes under alternative operational abstention rules

The controlled MNAR diagnostic above measures credit-recovery sensitivity; it is not an end-to-end training experiment under naturally induced MNAR mechanisms. We separately compared three implementable handling rules on the same three training seeds, with other settings fixed:

  • Default calibrated fallback: Use calibrated soft credit when an audit abstains — 28.3% resolved rate.
  • Pessimistic failure assignment: Treat a crashed, timed-out, or indeterminate audit leaf as failure — 28.1%.
  • Drop the credit unit: Remove the entire unit if any of its leaves abstains — 27.6%.

The 0.2-point difference between calibrated fallback and pessimistic assignment is too small to support a strong statistical ranking by itself. Dropping abstaining units performs worse in this setting, consistent with a potential survivor-selection problem. Taken together, these results show that (i) controlled MNAR deviations impair credit recovery but leave some useful recovery in the tested range and (ii) calibrated fallback performs best among the tested online handling rules. Neither result alone identifies end-to-end performance under the unknown natural MNAR mechanism, and neither makes Equation (9) an unconditional unbiasedness guarantee.

AuthorsOpenReview ↗

Testing Informative Audit Abstention, Compute Fairness, and Reference-Verdict Quality

3. Equal GPU-hours: compute overhead does not eliminate the measured advantage

We agree that the original token and sandbox caps did not equalize total GPU cost. Table 10 reports 318.4 node GPU-hours for BANC, versus 291.2 for BPO+DIFF, an increase of approximately 9.3%. We will state this explicitly and provide a supplementary resource ledger (Appendix A.8) covering generated tokens, sandbox seconds, and GPU-hours for the original baseline, equal-audit-access baseline, extended-GPU baseline, and BANC.

We evaluated both directions of GPU-budget matching. Extending BPO+DIFF to 318.4 node GPU-hours raises its resolved rate from 25.6% to 25.9%, still 2.4 points below full BANC at 28.3%. Stopping BANC at 291.2 node GPU-hours yields 27.9%, still 2.3 points above the original BPO+DIFF result at that GPU allowance.

As noted in the response to Rmjz, this is a separate equal-GPU-hour sensitivity analysis. The baseline may exceed the original 8.0×1078.0\times10^7-token limit to use its additional compute, so it should not be mistaken for simultaneous equality of all budget dimensions. Within the tested range, simply allocating the GPU-hour difference to additional baseline training does not close the measured gap. BANC nevertheless incurs higher total training GPU cost; our efficiency claim is limited to higher resolution under the original token/sandbox caps and lower sandbox usage.

Reviewer_RmjzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

BANC treats executable feedback for coding agents as costly, noisy, and selectively observed measurements. It randomizes branch and audit purchases under token/sandbox budgets, fits reliability models using inverse audit propensities, and constructs local action-unit credit with AIPW. The method combines active acquisition with selection correction to avoid changing the estimand by observing only promising branch points.

Critical strengths
  1. Which feedback is worth purchasing and how to prevent purchase preferences from changing the learning target are related but distinct questions. Randomization, positive propensity floors, exact logging, and cross-fitting connect them in an analyzable training pipeline.
  2. The diagnostic set, noise stress test, adaptive-masking experiment, and 232^3 factorial ablation probe different mechanisms.
Critical weaknesses
  1. Equation (9) conditions on the full acquisition history Fi\mathcal F_i and the joint event Di=1D_i=1 that all leaves are determinate. Assumption (iii), however, requires mean calibration only given (yℓ,wℓ)(y_\ell,w_\ell). Calibration on a coarser feature set does not automatically survive conditioning on richer history or joint determinacy.
  2. AIPW corrects branch selection, but not conditional measurement bias by itself.
  3. Matching token and sandbox caps is meaningful, but it is not matching total compute. Table 10 reports 318.4 GPU-hours for BANC versus 284.8 for APPO and 291.2 for BPO+DIFF, increases of approximately 11.8% and 9.3%. Wall-clock time also increases by 4.2 hours over APPO. The paper should distinguish improved performance with lower sandbox usage from overall training efficiency.
Questions for authors

What assumption ensures that calibration on (y,w)(y,w) remains valid after conditioning on Fi,Di=1\mathcal F_i,D_i=1? Please state it and complete the conditioning argument.

8 comments
AuthorsOpenReview ↗

Repairing the Conditional Calibration Argument and Accounting for Total Compute

We thank the reviewer for identifying two consequential distinctions: calibration on (yℓ,wℓ)(y_\ell,w_\ell) need not survive conditioning on the full acquisition history and joint determinacy, and matched token/sandbox caps are not matched training compute. We agree with both criticisms. Below we make the estimator's target and sufficient assumptions explicit, separate measurement bias from selection bias, and report a separate equal-GPU-hour comparison.

1. AIPW corrects acquisition selection—not misspecification of measured credit

This is the main gap in the original argument for Equation (9). Equation (1) fits a reliability model with the relatively coarse conditioning variables (yℓ,wℓ)(y_\ell,w_\ell). That does not imply correctness after additionally conditioning on the complete purchase history Fi\mathcal F_i, variables used for audit selection, or the joint event that every leaf is determinate. AIPW can correct which branch measurements were observed under randomized acquisition and overlap; it does not automatically turn a biased measured pseudo-credit into the target reference credit.

To distinguish potential from observed events, fix the following potential measurement protocol. For each eligible unit ii, imagine purchasing a branch under frozen πθold\pi_{\theta_{\mathrm{old}}}, generating KK sibling continuations, and applying the prespecified randomized audit mechanism. Denote the resulting potential calibrated pseudo-credit by Δ~i(1)\widetilde\Delta_i^{(1)}. Let Di(1)=1D_i^{(1)}=1 mean that all five leaves in the online K=4K=4 protocol would receive determinate verdicts under the full reference procedure. Both quantities are well-defined as potential outcomes even for units that were not actually branched; we do not claim that they are observed in the ordinary training log. The observed diagnostic event DiobsD_i^{\mathrm{obs}} is distinguished from this latent condition.

Let piB=Pr⁡(SiB=1∣Fi)p_i^B=\Pr(S_i^B=1\mid\mathcal F_i) be the true, exactly logged branch propensity, and let mi−m_i^- be fixed before acquisition or obtained through cross-fitting. The AIPW construction is

=m_i^-+\frac{S_i^B}{p_i^B} \bigl(\widetilde\Delta_i^{(1)}-m_i^-\bigr).$$ If $S_i^B$ is conditionally independent of the **joint potential outcome** $(\widetilde\Delta_i^{(1)},D_i^{(1)})$ given $\mathcal F_i$, and $p_i^B\ge p_{\min}^B>0$, then $$\mathbb E\!\left[\widehat\Delta_i^{AC}\mid\mathcal F_i,D_i^{(1)}=1\right] =\mathbb E\!\left[\widetilde\Delta_i^{(1)}\mid\mathcal F_i,D_i^{(1)}=1\right]. \tag{R1}$$ **Equation (R1) removes bias from branch purchase selection alone** and does not require a perfectly specified $m_i^-$. If audit selection depends on further observable information, its conditional randomization must be formulated with respect to the *actual audit history*; a propensity or ignorability assumption stated only in terms of $(y_\ell,w_\ell)$ cannot replace that mechanism.
AuthorsOpenReview ↗

Repairing the Conditional Calibration Argument and Accounting for Total Compute

2. Corrected Equation (9): conditioning sufficiency and an explicit measurement-bias term

The operational reference credit is Δiref=zi0ref−1K∑k=1Kzikref.\Delta_i^{ref}=z_{i0}^{ref}-\frac{1}{K}\sum_{k=1}^K z_{ik}^{ref}. To connect (R1) to this reference quantity, we must separately bound calibration error on leaves whose audits are not purchased. Let HiℓA\mathcal H_{i\ell}^{A} contain all information available before the audit decision, including (yℓ,wℓ)(y_\ell,w_\ell), branch-acquisition history, gradient sensitivity, resource shadow prices, and any other features actually used by the audit policy.

First define leaf-level and joint potential determinacy:

=\mathbf 1\{\text{leaf }\ell\text{ is determinate under the reference protocol}\}, \qquad D_i^{(1)}=\prod_{\ell=0}^{K}D_{i\ell}^{(1)}.$$ As Sec. 3.2 of the original paper specifies, the calibrator targets reliability over the **population of observed leaves**, not a hypothetical marginal population of all potential sibling leaves. We therefore write $$q_{\mathrm{obs}}(y,w) :=\Pr\bigl(r=1\mid y,w,\ \ell\text{ observed},\ D_{i\ell}^{(1)}=1\bigr).$$ Let $\epsilon_q$ denote the maximum conditional absolute error of the lagged calibrator $q_-$ relative to $q_{\mathrm{obs}}$, and define the possible failure of conditioning sufficiency as $$\epsilon_{\mathrm{suff}} =\sup_{i,\ell} \Bigl| q_{\mathrm{obs}}(y_{i\ell},w_{i\ell}) -\Pr\bigl(r_{i\ell}=1\mid\mathcal H_{i\ell}^{A},D_i^{(1)}=1\bigr) \Bigr|. \tag{R2}$$ The substantive condition is that this discrepancy is **controlled at a stated tolerance**, $\epsilon_{\mathrm{suff}}\le\delta_{\mathrm{suff}}$, on the relevant positive-probability support—not merely that the difference of two probabilities is trivially bounded by one. It is an **additional assumption**, not a consequence of fitting a calibrator on coarse features. We also require randomized audit purchase conditional on $\mathcal H_{i\ell}^{A}$, the actual logged probability $p_{i\ell}^{A}$ to be used for sampling, and $p_{i\ell}^{A}\ge p_{\min}^{A}>0$. If audit randomization is conditionally independent of the potential reference label and joint determinacy given the complete audit history, and the above calibration and sufficiency errors are controlled for every potentially determinate leaf, the conditional bias per leaf is at most $(1-p_{\min}^{A})(\epsilon_q+\epsilon_{\mathrm{suff}})$. Since unit credit is the original leaf minus the mean of $K$ siblings, the two absolute-error contributions give $$\begin{aligned} &\mathbb E\left[\widehat\Delta_i^{AC}-\Delta_i^{ref} \mid\mathcal F_i,D_i^{(1)}=1\right]\\ &=\underbrace{\mathbb E[\widehat\Delta_i^{AC}-\widetilde\Delta_i^{(1)} \mid\mathcal F_i,D_i^{(1)}=1]}_{=0\;\text{(branch-selection term)}}\\ &\quad+\underbrace{\mathbb E[\widetilde\Delta_i^{(1)}-\Delta_i^{ref} \mid\mathcal F_i,D_i^{(1)}=1]}_{\text{measurement and conditional-calibration term}}. \end{aligned}$$ Consequently, under these conditions, $$\boxed{ \left| \mathbb E[\widehat\Delta_i^{AC}-\Delta_i^{ref} \mid\mathcal F_i,D_i^{(1)}=1] \right| \le 2(1-p_{\min}^{A})(\epsilon_q+\epsilon_{\mathrm{suff}}) }. \tag{R3}$$ **This is a sufficient-condition bias bound for the potentially fully determinate population, not unconditional unbiasedness over all training units.** The factor two comes from the original leaf and the average sibling contribution. When $D_i^{(1)}=0$, at least one binary reference verdict is undefined under the fixed protocol, so the same $\Delta_i^{ref}$ is not identified for that unit. One must either define a *different operational estimand*—for example, one that uses a prespecified calibrated fallback after ABSTAIN—or obtain an additional independent adjudication procedure. AIPW can address missing purchases relative to that newly specified target, but cannot identify reference labels that the protocol itself cannot decide. Appendix B.4 will state the potential outcomes, conditioning assumptions, and the two-part bias decomposition explicitly, replacing the original step from coarse calibration directly to Equation (9).
AuthorsOpenReview ↗

*Repairing the Conditional Calibration Argument and Accounting for Total Compute*

3. Empirical assessment of conditioning sufficiency—without treating it as a proof

The 41-dimensional feature set in Appendix A.3 already includes raw verifier outputs, execution time, exit codes, crash information, and some pre-acquisition unit features. These variables help, but they do not rule out residual dependence on fuller history or sibling outcomes.

We therefore analyzed the fully observed diagnostic set with N=512N=512 and K=32K=32, containing 512(32+1)=16,896512(32+1)=16{,}896 leaves with determinate reference verdicts. Under five-fold, task-level cross-fitting, we compared the original q−(yℓ,wℓ)q_-(y_\ell,w_\ell) against an augmented calibrator qaugq_{\mathrm{aug}}. The latter adds a 32-dimensional compressed policy-history representation and other sibling leaves' mean pass rates and runtime summaries. On held-out folds, the mean absolute prediction difference is ϵ^suffproxy=0.0074,\widehat\epsilon_{\mathrm{suff}}^{\mathrm{proxy}}=0.0074, with task-bootstrap standard error 0.0018.

This is an empirical proxy for sensitivity to additional features, not a mathematical upper bound on ϵsuff\epsilon_{\mathrm{suff}}. The augmented model's complete sibling information also need not be available in online K=4K=4 training. Two models may agree while both being miscalibrated, so 0.0074 cannot be substituted into (R3) as a certified sufficiency bound or used to claim unbiasedness in the online distribution. Its role is to show how strongly these measured additional covariates affect held-out reliability predictions.

4. Total compute: an explicit cost, and a separate equal-GPU-hour comparison

The reviewer is correct that our original resource comparison matched generated tokens and sandbox time, but not total training GPU compute. Appendix Table 10 reports:

MethodNode GPU-hoursTraining wall-clock
APPO284.835.6 h
BPO+DIFF291.236.4 h
BANC318.439.8 h

BANC uses 11.8% more node GPU-hours than APPO and 9.3% more than BPO+DIFF. Its reported 7.4% sandbox-time saving must therefore not be interpreted as lower total training compute.

We added two complementary equal-GPU-hour sensitivity comparisons:

  1. Extend the baselines to BANC's 318.4 node GPU-hours. BPO+DIFF increases from 25.6% to 25.9% resolved rate; APPO increases from 24.7% to 25.1%. BANC remains at 28.3%, leading the extended baselines by 2.4 and 3.2 points, respectively.
  2. Stop BANC at BPO+DIFF's 291.2 node GPU-hours. BANC reaches 27.9%, versus 25.6% for BPO+DIFF at the same GPU-hour allowance, a +2.3-point difference. At that checkpoint BANC uses approximately 1.01×1061.01\times10^6 sandbox seconds, or 15.1% less than BPO+DIFF's 1.192×1061.192\times10^6 seconds.

These are not comparisons simultaneously matched on every resource. To allow extended baseline training, the token budget may exceed the original 8.0×1078.0\times10^7 cap. We will report realized tokens, sandbox time, and GPU-hours for each such configuration in a separate ledger. The result indicates that the observed performance gap is not eliminated by spending the additional GPU-hour allowance on longer baseline training in the tested range. It does not negate the cost of BANC's online scoring, nor does it establish that BANC uses less total compute.

5. Revised scope of the contribution

The theoretical and empirical claims are now separated more precisely:

First, BANC provides an analyzable correction for adaptive branch-selection bias; the relationship of its pseudo-credit to reference credit still requires explicit assumptions about conditional reliability and abstention.

Second, BANC improves issue resolution under the original generated-token and sandbox constraints, at a moderate increase in total GPU compute. Its benefit is better allocation and use of execution feedback, not a universal reduction in training cost.

We appreciate the reviewer's identification of both boundary conditions. They strengthen the paper by making the estimator statement and the resource-efficiency claim correspond to what the evidence actually supports.

Reviewer_whyWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BANC, a training-time layer for coding-agent RL that treats executable feedback as a budgeted and noisy measurement. BANC decides which branch and audit measurements to purchase based on their expected effect on the policy gradient, subject to shadow-priced token and sandbox costs. All purchases are randomized with logged probabilities. Raw verifier verdicts are calibrated against a higher-fidelity reference protocol, with inverse audit propensities correcting for the bias of adaptive auditing. Action-level credit is then formed with an augmented inverse-propensity estimator, so that adaptive acquisition does not change the credit being estimated. Each mechanism is first tested in isolation, and the full method is then evaluated end to end. With Qwen2.5-Coder-7B under matched token and sandbox caps, BANC improves resolved rate on a temporally held-out SWE-rebench suite by 2.7 points over BPO+DIFF, while using less sandbox time than APPO. The gains are concentrated on long and costly tasks and carry over to 16B and 32B backbones.

Critical strengths
  • Viewing executable feedback as a selectively acquired, noisy measurement is a useful perspective. It identifies a real problem in branching methods: heuristic branch selection and imperfect verifiers both bias credit, and prior work ignores this.
  • The evaluation is careful. Each mechanism is validated separately, the factorial ablation shows how the components interact, and the main comparison uses matched budgets, five seeds, and a prespecified confirmatory test.
Critical weaknesses
  • The individual components, including active testing, propensity-weighted calibration, AIPW estimation, and budgeted acquisition with dual prices, are established techniques, as the authors acknowledge. The contribution lies mainly in combining them for coding-agent RL. The paper would benefit from a clearer statement of what is technically new beyond this integration.
  • The method depends heavily on the reference protocol. Only 2,184 of 7,941 candidate tasks support it, so all methods are trained on a filtered subset that may favor well-tested repositories. It is unclear how BANC performs on tasks where property synthesis or mutation testing is not available.
  • The comparison with baselines may not be fully balanced. BANC has access to disjoint training-time test suites through audits, while BPO+DIFF and APPO rely only on the raw verifier. Part of the gain may therefore come from additional verification signal, not from the acquisition and correction framework itself.
  • The added complexity is substantial relative to the gain. BANC introduces three auxiliary models, a dual controller, and cross-fitting, and requires more GPU time and longer wall-clock training. In addition, the mean branch probability (0.029) sits close to the floor (0.025), which raises the question of how much the adaptive acquisition actually contributes in the main runs.
Questions for authors
  • How do BPO+DIFF and APPO perform when they are given the same audit access to the reference protocol?
  • How does BANC perform on tasks where the reference protocol cannot be constructed, or when training uses the unfiltered task pool?
  • Given that most branch probabilities stay near p_min, how different is the realized branch set from uniform sampling in the main runs?
  • How sensitive are the results to p_min, τ_acq, and the choice of audit recipe?
7 comments
AuthorsOpenReview ↗

Separating Audit Access from BANC's Gains and Evaluating Less-Filtered Training Pools

We appreciate the reviewer's recognition of the measurement perspective, mechanism-level diagnostics, and budget-controlled evaluation. We agree that the central attribution question is whether BANC's gains come from access to a stronger reference verifier, or from the acquisition and selection-correction framework itself. We therefore added matched-audit-access baselines, experiments outside the original fully auditable task pool, and analyses of actual acquisition behavior and compute costs.

1. BANC retains an advantage when baselines receive the same audit access

The original comparison did not fully isolate the information advantage conferred by reference audits. BANC could purchase high-fidelity audits, whereas the original BPO+DIFF and APPO baselines used only the raw verifier. Matching token and sandbox limits was necessary but did not equalize the kind of feedback available.

We therefore introduced BPO+DIFF + Ref-Audit and APPO + Ref-Audit. Both have access to the same reference protocol and a 0.080×1060.080\times10^6-second audit budget, and use a lightweight calibrator of the same architecture for unaudited leaves. Their original branching and credit-update rules are otherwise preserved.

All configurations retain the same hard caps of 8.0×1078.0\times10^7 generated tokens and 1.20×1061.20\times10^6 sandbox seconds. These are resource ceilings, not identical realized usage. The audit-augmented baselines nearly exhaust the sandbox ceiling: BPO+DIFF + Ref-Audit uses 1.198×1061.198\times10^6 seconds and APPO + Ref-Audit uses 1.200×1061.200\times10^6, accommodating audits by reducing the number of backbone trajectories and branch executions. In comparison, BANC uses 1.108×1061.108\times10^6 seconds under the same ceiling; no additional sandbox budget is granted.

MethodReference-audit accessSWE-rebench resolved rateSandbox time (10610^6 s)
APPONo24.7%1.196
APPO + Ref-AuditYes25.5%1.200
BPO+DIFFNo25.6%1.192
BPO+DIFF + Ref-AuditYes26.3%1.198
BANCYes28.3%1.108

Giving BPO+DIFF the reference audit improves it from 25.6% to 26.3%: stronger verification information explains part, but not all, of the original performance difference. With access and audit budgets equalized, BANC still leads BPO+DIFF + Ref-Audit by 2.0 percentage points while using about 7.5% less sandbox time.

This comparison does not, by itself, attribute the entire residual gap to one component. The methods still differ in audit selection, branch acquisition, and credit estimation. We interpret it jointly with the existing factorial ablation in Table 3 and the branch-versus-audit GVA split reported to DP11. The narrower, directly supported conclusion is that equalizing the audit interface and budget does not erase BANC's advantage.

AuthorsOpenReview ↗

Separating Audit Access from BANC's Gains and Evaluating Less-Filtered Training Pools*

2. What happens when the complete reference protocol is unavailable?

We agree that filtering to fully auditable tasks restricts the scope of the original results. Of 7,941 candidate training tasks, only 2,184 support the complete property- and mutation-testing reference protocol reliably. This improves consistency for mechanism diagnostics but may overrepresent repositories with stronger test infrastructure.

To test this limitation, we used the configurable AUDIT interface to introduce a lightweight environmental recipe, renvr_{\mathrm{env}}, based on clean-container reconstruction, diagnostic reruns, runtime invariants, exit codes, and crash checks. It does not require synthesized properties or mutation testing. On the original 500-leaf audit sample, it identifies 36 of the 59 raw/reference disagreements (61.0%), indicating a weaker yet informative audit signal.

We evaluated two expanded-pool settings:

  1. Training on a usable subset of the 5,757 tasks excluded by the final audit-availability filter. With three seeds, lightweight-audit BANC resolves 26.9% of held-out SWE-rebench tasks versus 24.7% for BPO+DIFF trained on the same pool, a +2.2-point difference. A variant retaining GVA and AIPW but disabling audit calibration reaches 25.8%.
  2. Mixed training using an expanded pool drawn from all 7,941 candidates. Under the same setting, BPO+DIFF reaches 26.1%, while BANC reaches 28.9%, a +2.8-point difference.

We clarify the filtering history because it determines what these results establish. The 5,757 excluded tasks are not failures of earlier reproducibility checks. As shown by the funnel in Appendix A.1/Table 4, the 7,941 candidates had already passed the preceding checks: reproducible build (15,912 remaining), fail-to-pass validation (9,847), raw-verifier completion within 900 seconds (8,516), cross-source deduplication (8,203), and exclusion of evaluation overlap (7,941). The final reduction from 7,941 to 2,184 required a complete audit protocol: at least five synthesized properties passing stably on the reference patch, and a mutation harness completing within its time limit. The expanded-pool experiments replace this final requirement with the weaker renvr_{\mathrm{env}} procedure; they do not redefine the earlier filters.

Because the original verifier and reconstructed environment are executable for these 7,941 candidates, renvr_{\mathrm{env}} can be attempted on all 5,757 previously excluded tasks. However, successful raw-verifier execution does not guarantee a determinate or high-quality audit: individual leaves may still crash, time out, or return ABSTAIN under the lighter diagnostic. We will report the effective audit coverage, abstention rate, resource usage, and breakdown of failures at the final filtering stage separately for each expanded setting.

Thus, BANC is not restricted in principle to repositories supporting the full property-testing reference, but performance depends on the reliability information supplied by the selected audit recipe. A lightweight environmental audit should not be regarded as equivalent to the complete reference protocol.

AuthorsOpenReview ↗

Separating Audit Access from BANC's Gains and Evaluating Less-Filtered Training Pools*

3. How different are the actual GVA purchases from uniform sampling?

The reviewer's concern about pˉB=0.029\bar p^B=0.029 versus pmin⁡=0.025p_{\min}=0.025 is well founded. As discussed in our response to DP11, the floor already uses 86.2% of expected branch-purchase mass. The appropriate interpretation is that GVA concentrates the limited remaining allocation where credit-relevant information is predicted to be most useful, not that it dramatically changes the full candidate distribution.

The highest-propensity 10% of online candidates receive 21.8% of expected branch purchases and account for approximately 95.3% of above-floor adaptive mass. Among purchases observed under each separately trained acquisition policy, complex cross-block edits and error-recovery actions represent 39.5% under uniform acquisition versus 52.4% under GVA; observed nonzero credit divergence occurs in 34.1% versus 48.6%, respectively.

These composition statistics are descriptive, not a fixed-candidate-pool causal experiment. The trained policy itself changes the distribution of eligible actions, so the entire 12.9-point composition shift cannot be attributed solely to reweighting pcp_c. Fixed-pool replay is needed for that attribution; the propensity distribution and expected top-decile purchase share directly characterize allocation over the logged candidate pool.

The mechanism-level result is consistent with targeted allocation: with calibration and AIPW held fixed, full GVA increases resolved rate from 26.8% to 28.3% and reduces normalized gradient error from 0.47 to 0.34 (Table 3). Splitting the two purchase decisions gives 27.5% with branch-only GVA, 27.6% with audit-only GVA, and 28.3% with both. The benefit is not contingent on a large average branch-propensity shift; both types of purchased measurements contribute.

Reviewer_DP11OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper recasts executable feedback for coding-agent RL as a purchase: branch measurements and verifier audits are costly, noisy, and acquired non-uniformly, so it proposes BANC, a training-time layer with three components. GVA scores each candidate purchase by its expected effect on the policy gradient, Û_i = G_i σ̂²_i q̂^pre_i, against shadow-priced resource costs under a multi-resource dual controller, with purchase probabilities p_c = p_min + (p_max − p_min)·σ((Û_c − λᵀb_c)/τ_acq) bounded in [p_min, p_max] = [0.025, 0.40] and logged (Appendix B.1). Calibration fits an inverse-propensity-weighted reliability model q_η over a 41-dimensional feature vector to correct trained verdicts, since audits are bought preferentially and the raw verifier is a biased label. AIPW credit forms an augmented inverse-propensity pseudo-outcome Δ̂^AC for action-level credit, claimed conditionally unbiased given overlap, randomization and calibration within ε_q (Sec. 3.4, Eq. 9, Appendix B.4).

Critical strengths

The observation that a coding agent's leaf verdicts are a selected, noisy sample rather than ground truth is right, and the paper instruments it rather than asserting it: a 500-leaf uniform audit reporting 12.1% disagreement against a stricter property-based protocol, broken down by cause (27 flaky executions, 19 timeouts, 13 incomplete assertions), plus a 1,024-leaf disjoint calibration set. This matters because the framing shift from "sparse reward" to "biased, costly measurement" changes what the field should be building, and the audit number is the kind of figure that other groups can check against their own harnesses. It is the paper's most transferable artifact.

Critical weaknesses

The treatment is assigned per training run, but Appendix A.5 defines d_t = (1/5)Σ_s(Y^BANC_{t,s} − Y^BPO+DIFF_{t,s}) and then Rademacher-sign-flips over the 640 tasks, averaging seed variance away before testing. Table 15's per-seed counts (BANC 184/178/185/177/182 vs BPO+DIFF 166/161/167/160/165) give five paired differences of +18/+17/+18/+17/+17, all positive with perfectly disjoint ranges — but with n = 5 the exact two-sided sign-flip/test returns 2/2⁵ = 0.0625, which is also the minimum attainable p for any exact two-sided test at that n. So the number in the abstract (0.019) is strictly smaller than the smallest value the design can produce at the level where the treatment was assigned. The claim as written is a claim about the method, and the design licenses only "consistent across these five runs." This is consequential because it is the sole prespecified confirmatory contrast, the other comparisons are labelled secondary, and no confidence interval on any between-method difference appears anywhere in the paper.

Questions for authors
  1. Please report the seed-level test explicitly: for the five paired per-seed differences in Table 15, the exact two-sided sign-flip / Wilcoxon p is 0.0625, which is also the attainable floor at n = 5. What do you claim, word for word, is the null being rejected at p = 0.019, and what population does it license inference over? If the intended claim is reproducibility of the method, please state that the design cannot reach α = 0.05 and either add seeds (with the observed 17–18/640 effect and a between-seed SD of ~0.55, roughly 6–7 concordant seeds are the minimum) or report a variance decomposition (seed vs task) with an interval on the difference.
  2. Report the full empirical distribution of logged p_c in the deployed runs: deciles, the fraction at p_min, the fraction above 0.1, and the achieved Spearman correlation between Û and realized credit reduction inside the training loop (not the 512-unit diagnostic). Explain the arithmetic of mean p_c = 0.029 against p_min = 0.025 and p_max = 0.40: is the sigmoid saturated at the floor for essentially all candidates? Then split the GVA flag into its two components — (branches under GVA, audits uniform) versus (branches uniform, audits under GVA) — since audits cost only 7% of sandbox time and calibration carries the largest main effect. Also sweep p_min ∈ {0.01, 0.025, 0.05, 0.10} and K ∈ {2, 4, 8} at fixed cost.
  3. For Eq. 9's guarantee: state the fraction of training updates in which some leaf of the unit would have abstained (D_i = 0), report the marginal estimand or a bound on the D_i = 0 contribution, and test assumption (iv) by comparing r_ℓ for abstaining vs determinate audits on common (y, w) strata — Appendix D.3's own example (a background daemon leaving a nonzero exit status) is exactly where abstention plausibly correlates with the verifier being wrong. The 2.4% leaf-level abstention rate is reported; the unit-level rate, needed for a K+1-leaf unit, is not.
  4. Position the contribution against the nearest prior rather than by transfer. Sec. 2 concedes the construction is a relabelling of doubly-robust estimation ("those estimators correct for which actions a logging policy took; BANC applies the same construction to which measurements were purchased") — which suggests the novelty is the acquisition score and the audit-utility coupling, not the view. Specifically: how does Eq. 3 differ from Ash et al. (2020)'s gradient lower bound for active learning (cited once in Sec. 2, never again) and from gradient-norm importance sampling; and how does sequential branch-then-audit acquisition with a composed propensity differ from budgeted counterfactual-annotation OPE? Please also state how BANC degrades when the detector is replaced by a learned judge rather than the hand-written property protocol.
11 comments
AuthorsOpenReview ↗

Clarifying Statistical Inference, Actual Acquisition Behavior, and Conditional Credit Guarantees

1. What does p=0.019p=0.019 test? Seed-level replication and uncertainty

We agree that the originally reported p=0.019p=0.019 does not establish statistical significance across independent training runs.

Appendix A.5 first forms, for each of the 640 evaluation tasks, a paired difference dtd_t averaged over the five trained seeds, and then applies a two-sided, studentized Rademacher sign-flip test over tasks. The null tested at p=0.019p=0.019 is the zero-centered sign-flip symmetry of those task-level paired differences, conditional on the five trained model pairs and the evaluation protocol. Tasks, not independent training runs, are the units of sign flipping and inference. The result concerns differences across the task distribution for those particular models; it cannot establish method-level reproducibility across new training runs. We will remove any wording that interprets this task-level pp-value as seed-level significance.

As the reviewer correctly observes, the five seed-wise differences in Table 15 are [18,17,18,17,17][18,17,18,17,17] resolved tasks. All are positive, but the minimum attainable two-sided exact sign-test pp-value with five seeds is 2/25=0.06252/2^5=0.0625. Thus, the original five runs show consistent improvements, not seed-level significance at α=0.05\alpha=0.05.

To address reproducibility directly, we added two independently trained seeds (3407 and 4096), following the same training and evaluation protocol. Their BANC versus BPO+DIFF resolved-task counts are, respectively, 183 vs. 166 and 180 vs. 162. The seven paired differences are [18,17,18,17,17,17,18].[18,17,18,17,17,17,18]. All seven are positive, yielding a two-sided exact seed-level sign-test value of p=2/27=0.015625p=2/2^7=0.015625 (≈0.0156\approx0.0156). This is an additional, exploratory replication analysis, not the originally prespecified confirmatory test. To prevent post hoc reinterpretation, we will report separately: the original five-seed task-level test (p=0.019p=0.019), the original five-seed exact sign test (p=0.0625p=0.0625), and the new seven-seed exact sign test (p=0.015625p=0.015625). The seven-seed mean improvement remains approximately +2.72 percentage points.

We also decomposed the paired task-by-seed outcomes from the original five seeds, distinguishing task variation from task-by-seed interactions. The estimated task component is approximately 0.0715, the residual interaction component approximately 0.0703, and the moment estimate for the global seed main effect truncates at zero. The resulting overall standard error is approximately 1.16 percentage points; a paired two-way bootstrap over tasks and seeds gives a 95% CI of [+0.45, +4.99] percentage points. These estimates concern the original five-seed matrix, not the seven-seed test. We will provide the complete paired matrix, variance decomposition, resampling algorithm, and reproduction code in Appendix A.5. The interval indicates a positive estimated average difference under this resampling model, but with only five independent training runs it must not be presented as an assumption-free seed-level interval.

AuthorsOpenReview ↗

Clarifying Statistical Inference, Actual Acquisition Behavior, and Conditional Credit Guarantees

2. Does pˉB=0.029\bar p^B=0.029 imply effectively uniform acquisition?

The adaptive portion of the acquisition budget is indeed small, but it is not uniformly distributed. With pmin⁡=0.025p_{\min}=0.025 and mean branch propensity pˉB=0.029\bar p^B=0.029, the overlap floor accounts for 0.025/0.029=86.2%0.025/0.029=86.2\% of expected purchase-probability mass; only 13.8% lies above the floor. GVA therefore performs targeted allocation within a heavily floor-constrained budget, rather than wholesale reprioritization of all candidates.

We aggregated the actual logged candidate propensities from deployed training. Branch-propensity percentiles P10–P60 are 0.0250; P70, P80, P90, P95, and P99 are 0.0252, 0.0258, 0.0280, 0.042, and 0.145. Audit propensities have P10–P50 at 0.0250, P60 at 0.0251, and P70, P80, P90, P95, and P99 at 0.0256, 0.0272, 0.0315, 0.064, and 0.225. Across the same logs, 64.2% of branch candidates and 58.5% of audit candidates have a recorded probability exactly equal to the quantized floor; 1.6% and 2.8%, respectively, have p>0.10p>0.10.

The full distribution is summarized below. Each decile mean is aggregated directly from candidate-level probabilities, not reconstructed from percentile cutoffs.

StatisticBranch GVAAudit GVA
P10–P500.02500.0250
P600.02500.0251
P700.02520.0256
P800.02580.0272
P900.02800.0315
P950.0420.064
P990.1450.225
D1–D6 mean0.02500.0250
D7 mean0.02510.0253
D8 mean0.02550.0264
D9 mean0.02630.0273
D10 mean0.06310.0950
Overall pˉ\bar p0.02900.0324
Fraction recorded at pmin⁡p_{\min}64.2%58.5%
Fraction with p>0.10p>0.101.6%2.8%
Top-decile expected purchase share 0.1D10/pˉ0.1D_{10}/\bar p21.8%29.3%
Top-decile share of above-floor mass 0.1(D10−pmin⁡)/(pˉ−pmin⁡)0.1(D_{10}-p_{\min})/(\bar p-p_{\min})95.3%94.6%
Online Spearman, observed-credit subset0.480.55

Here D1–D10 are equal-frequency bins ranked by final purchase propensity, which incorporates resource shadow prices and therefore need not follow the ranking of unadjusted gradient utility. The decile means recover the overall means via 0.1∑k=110Dk0.1\sum_{k=1}^{10}D_k, up to displayed rounding. From the reported D10 means, the top decile accounts for 0.1(0.0631)/0.0290=21.8%0.1(0.0631)/0.0290=21.8\% of expected branch purchases and 0.1(0.0950)/0.0324=29.3%0.1(0.0950)/0.0324=29.3\% of expected audits. It captures approximately 95.3% and 94.6% of above-floor probability mass, respectively. This is concentrated allocation of a small adaptive margin, not a claim of large overall departures from uniformity.

The apparent mass exactly at pmin⁡p_{\min} is also an implementation detail that we will make explicit. The real-valued sigmoid output praw=pmin⁡+(pmax⁡−pmin⁡)σ(z)p_{\mathrm{raw}}=p_{\min}+(p_{\max}-p_{\min})\sigma(z) is strictly above pmin⁡p_{\min}. The implementation quantizes the final probability to four decimal places before Bernoulli sampling, samples using that quantized value psamplep_{\mathrm{sample}}, and logs the identical value plogged=psamplep_{\mathrm{logged}}=p_{\mathrm{sample}}. AIPW uses this actual logged sampling propensity, not an unquantized proxy. Thus, raw probabilities in [pmin⁡,pmin⁡+0.00005)[p_{\min},p_{\min}+0.00005) are recorded as 0.0250. All percentiles, decile means, and floor fractions above come from this one quantized log. For audits, the floor fraction is 58.5%, so P60 has risen to 0.0251 even though D6 (the 50th–60th percentile bin) still rounds to a mean of 0.0250. For branches, the 64.2% floor fraction makes both P60 and D6 equal to 0.0250. We will release the quantization, sampling, and logging logic alongside the corresponding appendix statistics.

AuthorsOpenReview ↗

Clarifying Statistical Inference, Actual Acquisition Behavior, and Conditional Credit Guarantees

The deployed utility score also carries predictive information beyond the offline 512-unit diagnostic. Within the online subset of purchased candidates for which post-purchase credit outcomes are observable, Spearman correlations between predicted pre-purchase utility and a proxy for subsequent credit-residual reduction are 0.48±0.020.48\pm0.02 for branches and 0.55±0.020.55\pm0.02 for audits. These are rankings conditional on observed purchases, not unbiased estimates over all eligible candidates; we will report the evaluation-set construction and sample sizes with the logs.

To separate branch and audit acquisition, we fixed calibration and AIPW on and varied only the two GVA switches:

Acquisition policySWE-rebench resolved rate
Uniform branches + uniform audits26.8%
GVA branches + uniform audits27.5%
Uniform branches + GVA audits27.6%
GVA branches + GVA audits28.3%

Full GVA improves upon uniform acquisition by +1.5 percentage points. Both stages contribute: the interaction 28.3−27.5−27.6+26.828.3-27.5-27.6+26.8 is approximately zero at the displayed precision, indicating approximately additive rather than demonstrably superadditive gains.

At fixed overall resources, the resolved rates for pmin⁡∈{0.01,0.025,0.05,0.10}p_{\min}\in\{0.01,0.025,0.05,0.10\} are 27.6%, 28.3%, 27.4%, 26.1%; for K∈{2,4,8}K\in\{2,4,8\} they are 27.4%, 28.3%, 28.1%. Increasing the propensity floor can consume so much of the fixed purchase budget that fewer backbone trajectories fit under the cap. These are therefore end-to-end allocation trade-offs, not isolated effects of a propensity parameter.

Finally, we agree that the original statement of an effective sample size of 0.74Nelig0.74N_{\mathrm{elig}} conflated distinct statistics and placed the number in an imprecise context in Sec. 4.3. We will distinguish (i) the conventional, inverse-propensity-weighted Kish ESS among purchased branches (approximately 1,556, averaged over the five online runs) from (ii) relative variance efficiency on the N=512N=512 fully observed diagnostic units:

\frac{\widehat{\operatorname{Var}}(\hat\mu_{\mathrm{full}})} {\widehat{\operatorname{Var}}(\hat\mu_{\mathrm{AIPW}})} \approx\frac{1}{1.35}\approx0.74.$$ Both variances are estimated by paired, task-clustered bootstrap resampling. The original $0.74N_{\mathrm{elig}}$ expression should be read as a scaled variance-efficiency description from this *diagnostic population*, **not as 0.74 times the eligible pool being independently or fully audited**, and not as an online Kish ESS. We will correct Sec. 4.3 to state “diagnostic relative variance efficiency: 0.74,” explicitly revise its original placement among online-run statistics, and provide the two estimated variances and resampling procedure in the appendix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

BANC: Clarifying Our Contribution and Addressing the Main Concerns

We thank all four reviewers for their careful evaluation and constructive comments. Their feedback helped us clarify BANC's contribution, test alternative explanations for its gains, and make the statistical and theoretical claims more precise.

The reviews highlighted three strengths:

  • A practically important problem. Executable feedback is costly, noisy, and selectively observed; its acquisition affects the learning signal.
  • A connected methodological design. Randomized, logged purchases connect feedback acquisition with correction of selection bias.
  • Evidence beyond benchmark scores. Credit-recovery diagnostics, verifier-noise experiments, and factorial ablations examine why the combined system improves.

Clarifying BANC's Contribution

BANC connects three decisions in coding-agent training: which branch or audit measurements to purchase, how much to trust their outcomes, and how to use selectively observed feedback for action-level credit.

These decisions are linked: preferential branching changes the observed credit distribution, while verifier noise can distort the measurements themselves. GVA combines gradient sensitivity, counterfactual uncertainty, and predicted reliability against resource costs; calibration and selection correction govern how the purchased feedback enters learning. IPW/AIPW and dual control are established tools. Our contribution is their coordinated use for measurement acquisition and credit estimation in coding-agent training.

The acquisition mechanism has a measurable role within this design. With calibration and AIPW fixed, replacing uniform purchases with GVA raises SWE-rebench resolution from 26.8% to 28.3%; branch-only and audit-only variants reach 27.5% and 27.6% (Table 24). Both stages contribute within the limited adaptive margin left by the propensity floor.

Addressing the Main Concerns

The revisions directly address the reviewers' questions about the source of the gains and the conclusions they support:

  • Persistent gains under matched audit access and separate GPU controls. Under the same reference-audit interface, audit allowance, and original token/sandbox ceilings, BANC reaches 28.3% versus 26.3% for BPO+DIFF + Ref-Audit, retaining a 2.0-point advantage (Table 21). BANC's original configuration uses 9.3% more GPU-hours than BPO+DIFF. In a separate comparison at 318.4 GPU-hours, extending BPO+DIFF yields 25.9%, versus 28.3% for BANC (Table 22). The equal-GPU study permits different token usage. Thus, matching audit access or extending baseline training does not close the respective measured gaps.

  • Explicit statistical scope and additional seed-level replication. The original p=0.019 is a task-level sign-flip test over 640 paired differences, each averaged across five trained model pairs; inference is conditional on those models. The original five-seed exact sign test gives p=0.0625. Two added independent seeds also favor BANC, giving positive gains in all seven pairs and an exploratory exact sign-test p=0.015625. Appendix A.5 reports this replication separately from the prespecified analysis.

  • Corrected, explicitly conditional credit guarantee. The revised argument for Eq. (9) separates correction of randomized branch-selection bias from recovery of reference credit. The latter additionally requires adequate calibration conditional on the full audit history and joint determinacy. The revised statement provides an explicit conditional bias bound under overlap, randomization, and calibration assumptions, restricted to units whose potential reference outcomes are all determinate (Sec. 3.4; Appendix B.4). Controlled sensitivity tests examine informative audit abstention, which remains a limitation of the guarantee.

  • Independent reference checks and broader task coverage. Adjudication of the 59 determinate disagreements supports the reference in 51, the raw verifier in 6, and is inconclusive in 2 (Table 20). The reference therefore remains an imperfect operational target. On the same usable subset of previously excluded training tasks, lightweight-audit BANC reaches 26.9%, versus 24.7% for BPO+DIFF across three seeds (Appendix C.3). This extends the evidence beyond the original fully auditable pool while retaining audit quality as an explicit condition.

We thank the reviewers for helping sharpen the paper. The separate controls preserve BANC's measured advantage, while the corrected guarantee makes its assumptions and limits explicit. We respectfully ask that the contribution be reassessed in light of these responses and revisions, and would welcome clarification of any remaining concern material to the assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.