- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
To address the sparse reward problem of RLVR in code generation tasks, this paper proposes ASTrace, which decomposes programs into AST units and performs fine-grained credit assignment via two mechanisms: Progress Credit and Failure Localization. The paper reports pass@1 evaluation results across six benchmarks using two model backbones, demonstrating convincing empirical performance.
Critical strengths
ASTrace divides credit assignment into two components: progress credit for groups in which all rollouts fail, and failure localization for groups in which some rollouts succeed. The former estimates the marginal testing benefit of restoring code units in a particular order, while the latter leverages execution traces to pinpoint failing units. The two mechanisms are designed to address different rollout-group scenarios, resulting in a clear and intuitive framework. Moreover, the credit signals are derived directly from executable programs and test outcomes, without relying on an additionally trained reward model or reference solutions. The experiments cover multiple code generation benchmarks, supplemented by competitive programming evaluations on OJBench and LCB-Pro. The results show that ASTrac improves the average pass@1 across all six benchmarks on both model backbones.
Critical weaknesses
- Progress Credit restores code units sequentially in a random order and uses the marginal gains from a single permutation to estimate credit. This single-sample estimate may have high variance, particularly when complex dependencies exist between code units.
- Failure Localization identifies code units based on the traceback or execution trace of failed tests and assigns a fixed negative credit. However, the location where an exception is reported during execution does not necessarily correspond to the root cause of the error. How does Failure Localization avoid incorrect attribution in cases involving collaborative errors across multiple code units or downstream exceptions caused by upstream errors?
- The method is evaluated on only Qwen3 and Qwen2.5 models. It would be beneficial to include newer models, such as Qwen3.5, to further validate the effectiveness and generalizability of ASTrace.
Questions for authors
Please refer to the weaknesses above.
4 comments
Dear Reviewer 13um,
Thank you again for your positive assessment and helpful questions. In our response, we:
- compared the single-order estimate against exact Shapley values.
- ran a swap analysis on 108 failing rollouts.
- trained GRPO and ASTrace from Qwen3.5-4B-Base, a newer base model.
We would like to know whether these results address your questions. We are happy to discuss anything further.
Best regards, The Authors
Thank you for your detailed responses and clarifications! I think this is a good paper, and will keep my rating of 3 (Weak Accept).
Thanks to Reviewer 13um
Thank you for your prompt reply. We are glad that we have addressed your concerns. Again, we sincerely appreciate your time and support.
Best wishes,
Authors
We thank Reviewer 13um for the positive assessment, in particular for finding the two-route framework "clear and intuitive" and the empirical results "convincing". We measured the fidelity of both credit signals directly and address each question below.
[W1. Variance of the single-permutation estimate]
[W2. Reported location vs. root cause]
[W3. Newer backbones]
We would be glad to discuss any remaining questions.