- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AnchorCredit, a credit assignment method designed to mitigate cross-component credit misassignment in tasks with multiple verifiable rewards. The method leverages the policy model’s attention from output anchors to preceding reasoning sentences to estimate sentence–component associations and assign component-specific advantages to relevant reasoning, without requiring additional relevance annotations or a separate relevance discriminator. To support research on credit assignment in multi-component, verifiable-reward settings, the authors construct ProductAttr, a benchmark comprising 7,200 real-world product image–text samples. Experiments demonstrate that AnchorCredit outperforms existing baselines on ProductAttr, with additional evaluations on the public Nutrition5k dataset providing further evidence of its effectiveness. The authors also commit to releasing the code and dataset upon acceptance.
Critical strengths
-
The paper addresses an important bottleneck in RLVR.
-
The proposed method is presented clearly, intuitively, and in a well-structured manner, making the overall framework, its key components, and their interactions easy to follow and understand. The paper coherently explains the methodological design and clearly motivates the role of each component within the broader approach. This clarity helps readers understand the underlying rationale and appreciate how the different components work together to achieve the intended objectives. Overall, the well-organized presentation enhances the accessibility of the proposed method and facilitates a thorough understanding of the approach.
Critical weaknesses
-
The paper would benefit from more detailed case studies and qualitative analyses of credit assignment quality. For example, visualizing the credit assigned to individual components and comparing these allocations with those produced by alternative methods would provide deeper insights into the effectiveness and interpretability of the proposed approach.
-
The paper would be strengthened by a more comprehensive analysis of the computational costs associated with AnchorCredit compared with competing methods. Although the proposed framework demonstrates promising performance, the additional overhead introduced may affect its practical applicability. This analysis is particularly important because some reported performance improvements over the baselines are marginal, making it necessary to establish whether these gains justify the additional computational resources. The authors should therefore report and compare metrics such as training time, GPU hours, and inference cost across AnchorCredit and the baseline methods. If the performance gains are modest relative to the additional computational costs, the practical advantages of AnchorCredit may be less substantial than the results suggest.
Questions for authors
Please refer to the Weaknesses section, as they are also my concerns and questions for the authors. Any additional clarifications would be welcome.
4 comments
W2. Training cost, GPU hours, and inference cost.
Training time. We measured AnchorCredit’s overhead relative to GRPO on Qwen3.5-4B with 8×A100 GPUs:
| Measurement | GRPO | AnchorCredit | Change |
|---|---|---|---|
| Steady-window time per step | 106.7 s | 144.1 s | +35% |
| Complete training job | 22.71 h | 27.16 h | +20% |
| Peak GPU memory per training process | 37.64 GiB | 37.73 GiB | +0.09 GiB |
The overhead comes from one no-gradient forward pass and the computation of sentence-level credit from attention. No additional backward pass is required. By comparison, RTT requires separately training a Qwen3.5-4B relevance discriminator and repeatedly invoking it during policy training. AnchorCredit uses the policy model’s own attention and requires no additional model.
Because the +35% steady-window step difference above includes policy differences in generated rollout lengths during training, we separately timed the additional no-gradient attention forward inside each AnchorCredit update on fixed responses to isolate its direct cost. Each length bin comprised three fixed-response runs with 50 measured updates each on eight A100 GPUs:
| Fixed-input length | Extra forward (s) | Full update wall time (s) | Forward share |
|---|---|---|---|
| Short | 1.748 | 14.349 | 12.18% |
| Medium | 2.167 | 16.690 | 12.98% |
| Long | 2.728 | 21.330 | 12.79% |
Across 450 measured updates, total forward time was 12.69% of total update time. The numerator uses the maximum measured model-forward time across participating workers. The denominator starts at advantage computation and ends after the actor update and synchronization; generation, verifier reward computation, and reference-model inference are timed separately. In the full 600-step training trajectory, the entire attribution prepass, including forward, routing, communication, and driver finalization, took 12,798.27 of 85,306.61 seconds of training-step wall time (15.00%).
GPU hours. Multiplying the recorded job wall time by eight A100 GPUs gives approximately 181.68 nominal GPU-hours for GRPO and 217.28 for AnchorCredit. These are job-clock estimates rather than measurements of device binding and release.
Inference cost. AnchorCredit changes only training-time credit assignment. The trained model has the same architecture as its GRPO counterpart and requires no additional inference computation. We compared the two trained models on the same L20 GPU at concurrency 16, using three decoding seeds and 200 items per seed:
| Metric | GRPO | AnchorCredit |
|---|---|---|
| Mean request latency | 9.39 s | 6.14 s |
| Mean output length | 442 tokens | 263 tokens |
| Time to first token | 0.40 s | 0.40 s |
| Total time for 200 requests | 135.6 s | 173.6 s |
The AnchorCredit model produces shorter outputs on average and has lower mean request latency. However, a few requests in each run reach the 8,192-token generation limit, increasing the total batch completion time. These measured differences reflect the models’ output-length distributions rather than an additional inference operation required by AnchorCredit.
Benefits under a common time budget. To assess whether the overhead is justified, we used the recorded 22.65-hour job clock to GRPO’s step-599 checkpoint as a shared budget. For each method, we selected the latest checkpoint saved within that budget, with checkpoints saved every 100 steps: step 599 for GRPO and step 400 for AnchorCredit, reached after 19.15 hours.
| Method | Checkpoint | Training time | Weight RMAE↓ | Dimension RMAE↓ | Fold F1↑ | Validity↑ |
|---|---|---|---|---|---|---|
| GRPO | Step 599 | 22.65 h | 30.86 | 22.48 | 89.55 | 99.88 |
| AnchorCredit | Step 400 | 19.15 h | 26.13 | 20.59 | 90.07 | 99.38 |
AnchorCredit’s step-400 checkpoint achieves weight and dimension RMAE approximately 4.7 and 1.9 points lower than GRPO’s final checkpoint, respectively. Thus, it achieves lower error with less training time in this comparison. At the final checkpoint (step 599), AnchorCredit reduces weight RMAE from 30.86 to 25.12 (an 18.6% relative error reduction) and dimension RMAE from 22.48 to 19.68 (a 12.5% relative reduction), and improves SGD Joint Goal Accuracy by 6.00 percentage points (a 24.3% relative error reduction; Table 2).
The measurement windows and complete results are provided in Appendix G.3 (Tables 22–25). We hope these additional analyses address your concerns and would be happy to clarify any remaining questions.
Follow-up on our response
Dear Reviewer,
We would like to briefly follow up on our response to your review. We would appreciate your feedback on whether our clarifications address your concerns and whether any important questions remain unresolved. We would be happy to provide further clarification.
Thank you for your time and consideration.
Follow-up to W2 on training cost with FlashAttention-2
Our submitted implementation ran an additional policy forward pass to recover attention for credit assignment. We extended FlashAttention-2 to save the attention rows needed for candidate anchors during generation [1]. The table reports mean step and phase times for GRPO, the submitted AnchorCredit implementation, and the new implementation. Each run used Qwen3.5-4B on eight A100 GPUs and generated 256 responses per step.
| Metric | GRPO | Submitted AnchorCredit | AnchorCredit + FlashAttention-2 export |
|---|---|---|---|
| Mean training step (s) | 157.3 | 124.3 | 164.0 |
| Generation (s/step) | 59.6 | 19.5 | 57.3 |
| Reference log probabilities and values (s/step) | 45.0 | 38.1 | 37.4 |
| Post-rollout attribution (s/step) | — | 19.3 | 8.6 |
| Policy training (s/step) | 44.7 | 39.8 | 51.5 |
| Output tokens/response | 542 | 290 | 463 |
| Output tokens/s during generation | 2,327 | 3,805 | 2,069 |
| Output tokens/s over the full step | 882 | 598 | 722 |
| Generation seconds per 1,000 output tokens | 0.43 | 0.26 | 0.48 |
| Full-step seconds per 1,000 output tokens | 1.13 | 1.67 | 1.38 |
The new implementation used zero additional full-model attribution passes during the measurement window. Post-rollout attribution took 8.6 s/step, down from 19.3 s/step in submitted AnchorCredit (−55.3%). Compared with GRPO, the new implementation took 4.3% longer per step and generated 14.6% fewer output tokens. Across all eight GPUs, full-step time per 1,000 output tokens was 1.38 s vs. 1.13 s for GRPO (+22.1%). We divide the total measured time by the total number of output tokens. Each run sampled its own responses, and this window measures a portion of training.
In a separate fixed-output test comparing two versions that both reuse rollout attention, moving row selection from CPU to GPU reduced generation time by a median of 5.56% across eight same-GPU pairs.
[1] AnchorCredit FlashAttention-2 Supplement (AnchorCredit_FlashAttention2_Supplement.zip). The anonymous archive contains the kernel patch, GPU selector, timing data, and a script to reproduce the reported summaries.
We thank the reviewer for recognizing the importance of the credit-assignment problem and the clarity of the presentation. We respond below with additional qualitative analysis and computational measurements.
W1. Credit-assignment case studies and qualitative analysis.
We have added a two-page qualitative case study and sentence-level credit visualization in Appendix F (Figure 3, pp. 21–22 of the revised paper), which reproduces all 15 recorded spans (
S00–S14) and the final generated JSON of a complete wireless-earbud training rollout verbatim, alongside each span's normalized anchor-attention shares (Weight, Dimensions, Foldability), gated candidate selections, and final advantages under GRPO, GDPO, and AnchorCredit (Full).In this rollout, the model predicts 50 g, 65 × 65 × 35 mm, and non-foldable, whereas the ground truth is 40 g, 125 × 116 × 24 mm, and non-foldable (so weight and foldability perform well relative to the sampled group, while dimensions are inaccurate). Holding the captured batch of 256 responses and rewards fixed, the table below provides a condensed summary grouped by span type (this is only a brief summary; the complete verbatim trajectory across all 15 spans
S00–S14and the generated JSON is presented in full in Appendix F, Figure 3, pp. 21–22 of the supplementary material):S00–S02: general item & single-unit setupS03–S05,S07–S09: e.g.,S04“approximately 60mm wide and 60mm deep”S06,S10: e.g.,S10“roughly 50 grams for a single unit”S11–S13: e.g.,S13“cannot be folded or rolled”As visualized in Figure 3, GRPO and GDPO broadcast a single positive response-level advantage (+1.623 and +0.715) to every sentence, inadvertently reinforcing the flawed 65 × 65 × 35 mm dimension derivation (
S03–S05,S07–S09). In contrast, AnchorCredit abstains and uses the response-level fallback on general introductory sentences where attention is diffuse (S00–S02), assigns negative credit (−0.301) specifically to the six dimension-reasoning spans, and preserves positive credit for the weight (+1.681) and foldability (+0.720) reasoning spans. Full values are logged training tensors, GRPO is the captured response-level baseline, and GDPO is recomputed on the same raw-reward batch using the paper's 0.4/0.4/0.2 weights, with each method's normalization and clipping applied.This qualitative pattern is also supported quantitatively across the dataset: in our offline replay of 8,992 reasoning sentences (800 responses across 50 items), gating and sign-aware selection directly resolve 469 sentences with conflicting-sign candidates, and in Section 5.5 (Tables 5–6), AnchorCredit achieves 76.50% / 73.62% multilabel association micro-F1 (versus 61.09% / 64.43% for keyword matching and 38.05% / 48.74% for gradient attribution) and 92.29% primary-component precision after gating.