- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies RVV assembly kernel generation using Qwen3-14B. It first applies supervised fine-tuning on the four-layer RvvCoT corpus, followed by GRPO with Decomposed Execution Verification (DEV). DEV assigns partial credit through operator-specific diagnostic tests. The authors report an average ER@10 of 59% on a benchmark covering 38 operator types and real-hardware performance results for seven operators on two boards.
Critical strengths
- The RVV setting poses concrete challenges. RvvBench distinguishes compilation success from functional correctness and checks each candidate under three VLEN settings. This directly tests whether generated code depends on a fixed vector length.
- The diagnostic examples make DEV easy to understand. Table 13 separates SGEMM coverage, loading, accumulation, and addressing checks, while Figure 10 illustrates relevant code failures. These examples show why two kernels that fail the full harness might still deserve different feedback.
Critical weaknesses
- The current comparisons do not isolate the additional benefit of DEV’s semantic decomposition. Figure 5 reports 59% average ER@10 with DEV, compared with 30% using a whole-kernel reward. This is an interesting gain, but the paper does not directly compare DEV with an ordinary test-pass-fraction reward. DEV’s operator-specific semantic tests are not identical to these approaches. The current evidence does not establish whether the benefit comes from that semantic decomposition or from providing finer-grained test feedback. This distinction is central to assessing the method’s contribution.
- Key data and experimental details behind the main gains are missing. Appendix B describes corpus construction and filtering, but the final corpus size and composition, the actual number of evaluation tasks, and the exact training settings behind the main results are not reported in sufficient detail. The 38 operators are operator types, not task counts, and Table 11 gives ranges for several hyperparameters. The main ablation also lacks reported independent training repetitions or uncertainty estimates. These omissions do not show that the results are wrong, but they make it difficult to assess the supervision and evaluation conditions behind the gains and their stability. Since the corpus and reward design are central contributions, these gaps materially affect my recommendation.
Questions for authors
- Clarification of the contribution: Do you have an existing comparison between DEV and an ordinary test-pass-fraction reward, or another analysis that separates their effects? Please clarify whether the starting checkpoint and the training and verification budgets are comparable, and what this evidence establishes about semantic decomposition itself.
- Clarification of the experimental setup: What are the per-layer sample counts, mixture weights, teacher identities, and training/validation/test split sizes for RvvCoT? How many shape–hardware tasks are in the main evaluation, and what exact training configurations produced the results in Table 1 and Figure 5?
- Clarification of result stability: How many independent training runs underlie the 59% and 30% results in Figure 5? Are there existing repeated-run results or uncertainty estimates showing how this gain varies across training runs?
2 comments
Official Comment by Reviewer mn2C
Thank you for the clarification. The additional checks concern implementation correctness, but they do not establish DEV’s learning benefit over ordinary test-pass-fraction rewards. It also remains unclear how the recovered configuration relates to the original 59% versus 30% results, and whether the reward correction affects that comparison. The new material therefore does not resolve the main concerns behind my review, and I maintain my original recommendation.
Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your feedback. Below, we respectfully provide our detailed responses to address your concerns.
Q1. DEV vs. ordinary test-pass-fraction rewards
Thank you for this important question. DEV’s distinguishing design is semantically structured execution feedback; its incremental learning benefit over ordinary dense feedback remains unestablished.
1. Methodological distinction.
Conventional test-pass-fraction rewards count successful end-to-end tests, whereas DEV evaluates executable semantic obligations. For RVV kernels, these obligations include output coverage, pointer stepping, vector loads, accumulation, and reductions. A candidate may satisfy an intermediate obligation while failing complete numerical verification. DEV can therefore distinguish failure modes that ordinary full-output tests treat identically.
2. Empirical evidence.
Our original experiments report:
Notably, binary-reward GRPO does not improve upon SFT, whereas DEV achieves +25 percentage points over SFT and +29 points over binary-reward GRPO. These results highlight the importance of our reward design.
3. Supplemental investigation and validation.
We corrected the supplemental reward implementation in a separate working copy while preserving DEV's diagnostic definitions and weights. Both comparator arms now use identical compilation shaping, differing only in the source of their progress scores.
In summary, DEV provides semantically structured execution feedback, and the original experiments report substantial gains over binary rewards.
Q2. Corpus and experimental reproducibility
Thank you for emphasizing reproducibility. We recovered exact supplemental configurations:
1. Structured supervision.
RvvCoT contains four complementary layers: ISA/ABI rules, verified kernels, failure repairs, and hardware-performance reasoning, addressing distinct challenges of RVV generation.
2. Recovered experimental details.
Overlength training targets were excluded rather than truncated, preserving complete target sequences.
Q3. Statistical stability across independent runs
Thanks for your emphasis on reliable performance estimates.
Thank you again for identifying these important distinctions. We hope the clarified contribution, explicit implementation findings, and narrower interpretation help you reassess the work. We have carefully considered all your suggestions and we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!