- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In the context of ultra-small functional models, localized errors can cause an entire function call to fail. A model might correctly select the intended function and provide most of the required arguments, yet still fail due to a single incorrect value, a missing parameter, or the inclusion of extraneous fields. Existing approaches—such as the CKD+Sim-RL framework—attempt to address this by employing a "similarity reward" mechanism. While this rewards partially correct decisions, it fails to provide specific guidance for correcting the erroneous components. Furthermore, generation groups rejected by the data collector are typically discarded and rendered unusable for reinforcement learning; however, this paper argues that such rejected groups still hold significant value. To leverage these discarded groups, the authors propose "Discarded-Group Counterfactual Repair" (DGCR). Building upon the CKD strategy from the STAR framework—while retaining the Sim-RL reward—the authors identify that existing methods cannot pinpoint which specific fields are incorrect. DGCR addresses this by utilizing rejected generation groups that meet certain criteria: it matches the predicted output against the reference output, modifies a single AST (Abstract Syntax Tree) unit, and observes whether the score improves. To ensure that any observed improvement stems from the modification itself—rather than from a change in how the predicted and reference outputs are matched—the authors incorporate a "verifier." This verifier confirms that the improvement is genuine and persists even when the matching relationship remains constant; only modifications that pass this check are used as training targets. Ultimately, the model learns from these refined segments, incorporating two types of signals: the original CKD+Sim-RL signal and an additional, weighted supervisory signal derived from DGCR. In my view, the DGCR method proposed in this paper holds practical significance. Extracting effective corrective supervision from generation groups—which were rejected by the RL collector yet still meet the necessary criteria—represents a commendable innovation. Furthermore, the repair pipeline designed in the paper—integrating structural edits supported by reference answers, gain verification via controlled-invocation matching, and segment-level supervision weighted by these gains—offers a sound approach to localized error correction. Finally, the paper’s experiments demonstrate that incorporating DGCR learning into the CKD+SimRL framework yields a measurable improvement in accuracy.
Critical strengths
The article is well organized, logically coherent, and highly readable.On the basis of CKD+SimRL, a method of increasing DGCR learning is proposed. Training the model through the collector abandonment group is a good innovation. The experiment has also verified that it has achieved certain improvements. At the same time, the local correction in the paper has good practical significance.
Critical weaknesses
The abstract mentions that DGCR offers advantages; however, it does not provide specific data to show the actual extent of this improvement.
Questions for authors
Of the items in the discarded group, how many can actually be converted into effective supervision? Given that the verifier in Table 3 omits both gain checks and matching checks, is it still possible to determine the value of controlled matching?
3 comments
Thank you for clarifying the recovery rate and the role of controlled matching. I maintain my rating of 3 (Weak acceptance), although the joint ablation does not isolate the downstream benefit of the matching check.
Response to Reviewer X2XQ: Isolating the Downstream Contribution of Controlled Matching
Thank you for this follow-up. To isolate the incremental downstream contribution of controlled matching and directly address your concern, we conducted a factorized verifier ablation under the same unit-weight protocol as Table 3. Across all three settings, we keep the reference-guided candidate generator, parsing procedure, token round-trip check, local compiler, candidate-selection order, and training configuration fixed, varying only which verification checks are active.
| Verification configuration | Gain check | Controlled matching | BFCL Overall (%) | ACEBench Summary (%) |
|---|---|---|---|---|
| Full verification | ✓ | ✓ | 51.58 | 53.80 |
| Gain-only | ✓ | ✗ | 50.81 | 53.00 |
| Syntactic-only | ✗ | ✗ | 46.18 | 49.20 |
The factorized results reveal two patterns. First, the positive-gain check accounts for the larger share of the improvement. Moving from syntactic-only selection to gain-only verification increases BFCL Overall by 4.63 percentage points and ACEBench Summary by 3.80 percentage points, corresponding to 85.7% and 82.6% of the observed point-estimate lift from syntactic-only to full verification. Second, adding controlled matching on top of gain-only verification provides an additional 0.77-point improvement on BFCL and 0.80-point improvement on ACEBench, accounting for the remaining 14.3% and 17.4% of the observed lift. These results show that controlled matching has a measurable conditional downstream contribution. The positive-gain check provides the main performance improvement, while controlled matching serves as a targeted safeguard against rematching-induced pseudo-gains.
We sincerely appreciate your recognition of our paper, as well as your careful feedback. We hope that this factorized ablation clarifies the contribution of controlled matching. If any questions or concerns remain, we would be very happy to address them.
Related Content in the Revised Manuscript. We have added the corresponding factorized analysis in Appendix D.2.
Thank you for your generous and encouraging assessment of our work. We are honored by your recognition of DGCR’s central motivation and practical value, and we appreciate your positive evaluation of the paper as a whole. Below, we clarify the recovery rate and the role of controlled matching, and explain how these points are addressed in the revised manuscript.
1. Numerical improvement in the abstract
We have added the numerical gains to the abstract of the revised manuscript so that readers can see the value of DGCR at a glance.
2. Recovery rate and the role of controlled matching
Among the groups discarded by the collector at 0.6B, roughly 18.2% can be converted into an accepted local repair. This percentage is computed over the entire rejected-group pool and therefore measures group-level recovery rather than the fraction of individual responses. It shows that the current construction can turn a substantive subset of discarded groups into learning signal.
The verifier comparison in Table 3 is a joint ablation. The complete verifier requires both positive-gain verification and assignment-consistency checks, whereas the syntactic-only arm removes both the gain check and the matching check. Table 3 therefore establishes the value of the complete verification scheme. Controlled matching targets a specific failure mode: after one call is edited, ordinary greedy matching may rematch other predictions and create an apparent score increase that did not come from the edited call. DGCR therefore compares the ordinary gain with the gain under the original assignment, requires the two gains to agree, and keeps the assignments of unedited calls unchanged. Appendix C.2 gives a concrete example in which rematching produces a positive total gain while the fixed-assignment gain is negative. In our leave-one-out audit from the frozen starting point, removing each gate separately selected the same nine compiled candidates from two groups, ruling out one known form of spurious credit. Controlled matching is not a new training configuration. It is a verifier check that prevents rematching-based pseudo-gains.
Related Content in the Revised Manuscript. The numerical abstract update is in the abstract. The recovery-rate and ground-truth injection results are in Section 4.4, while the joint verifier ablation is in Section 4.3 and Table 3. The concrete rematching example is in Appendix C.2.