acceptodds
Desk rejected at ICLR 2027

AnchorCredit: Output-Anchored Credit Assignment for Structured Generation with Multiple Verifiable Rewards

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

In Reinforcement Learning with Verifiable Rewards (RLVR), Group Relative Policy Optimization (GRPO) and related methods typically compute group-relative advantages from response rewards and broadcast the same advantage to every generated token in a response to guide policy updates. However, we identify a systematic problem of cross-component credit misassignment in tasks with multiple verifiable results, such as joint multi-attribute prediction: low rewards for some components can penalize a well-performing component and its associated reasoning. To address this problem, we propose AnchorCredit. The method uses the policy model's attention from output anchors to preceding reasoning sentences to estimate sentence–component associations and assign component-specific advantages to the relevant reasoning, without additional relevance annotations or a separate relevance discriminator. To support the study of credit assignment with multiple verifiable rewards, we construct ProductAttr, a benchmark of 7,200 real product image–text samples. Experiments show that AnchorCredit outperforms existing baselines on this benchmark. Experiments on the public Nutrition5k dataset further support the method's effectiveness in this setting. The complete code and dataset will be released upon acceptance.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, credit assignment, structured generation, multi-reward optimization, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tFpAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AnchorCredit, a credit assignment method designed to mitigate cross-component credit misassignment in tasks with multiple verifiable rewards. The method leverages the policy model’s attention from output anchors to preceding reasoning sentences to estimate sentence–component associations and assign component-specific advantages to relevant reasoning, without requiring additional relevance annotations or a separate relevance discriminator. To support research on credit assignment in multi-component, verifiable-reward settings, the authors construct ProductAttr, a benchmark comprising 7,200 real-world product image–text samples. Experiments demonstrate that AnchorCredit outperforms existing baselines on ProductAttr, with additional evaluations on the public Nutrition5k dataset providing further evidence of its effectiveness. The authors also commit to releasing the code and dataset upon acceptance.

Critical strengths
  1. The paper addresses an important bottleneck in RLVR.

  2. The proposed method is presented clearly, intuitively, and in a well-structured manner, making the overall framework, its key components, and their interactions easy to follow and understand. The paper coherently explains the methodological design and clearly motivates the role of each component within the broader approach. This clarity helps readers understand the underlying rationale and appreciate how the different components work together to achieve the intended objectives. Overall, the well-organized presentation enhances the accessibility of the proposed method and facilitates a thorough understanding of the approach.

Critical weaknesses
  1. The paper would benefit from more detailed case studies and qualitative analyses of credit assignment quality. For example, visualizing the credit assigned to individual components and comparing these allocations with those produced by alternative methods would provide deeper insights into the effectiveness and interpretability of the proposed approach.

  2. The paper would be strengthened by a more comprehensive analysis of the computational costs associated with AnchorCredit compared with competing methods. Although the proposed framework demonstrates promising performance, the additional overhead introduced may affect its practical applicability. This analysis is particularly important because some reported performance improvements over the baselines are marginal, making it necessary to establish whether these gains justify the additional computational resources. The authors should therefore report and compare metrics such as training time, GPU hours, and inference cost across AnchorCredit and the baseline methods. If the performance gains are modest relative to the additional computational costs, the practical advantages of AnchorCredit may be less substantial than the results suggest.

Questions for authors

Please refer to the Weaknesses section, as they are also my concerns and questions for the authors. Any additional clarifications would be welcome.

4 comments

We thank the reviewer for recognizing the importance of the credit-assignment problem and the clarity of the presentation. We respond below with additional qualitative analysis and computational measurements.

W1. Credit-assignment case studies and qualitative analysis.

We have added a two-page qualitative case study and sentence-level credit visualization in Appendix F (Figure 3, pp. 21–22 of the revised paper), which reproduces all 15 recorded spans (S00–S14) and the final generated JSON of a complete wireless-earbud training rollout verbatim, alongside each span's normalized anchor-attention shares (Weight, Dimensions, Foldability), gated candidate selections, and final advantages under GRPO, GDPO, and AnchorCredit (Full).

In this rollout, the model predicts 50 g, 65 × 65 × 35 mm, and non-foldable, whereas the ground truth is 40 g, 125 × 116 × 24 mm, and non-foldable (so weight and foldability perform well relative to the sampled group, while dimensions are inaccurate). Holding the captured batch of 256 responses and rewards fixed, the table below provides a condensed summary grouped by span type (this is only a brief summary; the complete verbatim trajectory across all 15 spans S00–S14 and the generated JSON is presented in full in Appendix F, Figure 3, pp. 21–22 of the supplementary material):

Reasoning span excerpt (from Figure 3, Appendix F)Largest anchor share (gated)GRPOGDPOFull (logged)
S00–S02: general item & single-unit setupNone pass gate (diffuse: 30–48%)+1.623+0.715+1.645 (fallback)
S03–S05, S07–S09: e.g., S04 “approximately 60mm wide and 60mm deep”Dimensions (55.1%–84.2%)+1.623+0.715−0.301
S06, S10: e.g., S10 “roughly 50 grams for a single unit”Weight (80.6%, 85.3%)+1.623+0.715+1.681
S11–S13: e.g., S13 “cannot be folded or rolled”Foldability (59.4%–85.4%)+1.623+0.715+0.720

As visualized in Figure 3, GRPO and GDPO broadcast a single positive response-level advantage (+1.623 and +0.715) to every sentence, inadvertently reinforcing the flawed 65 × 65 × 35 mm dimension derivation (S03–S05, S07–S09). In contrast, AnchorCredit abstains and uses the response-level fallback on general introductory sentences where attention is diffuse (S00–S02), assigns negative credit (−0.301) specifically to the six dimension-reasoning spans, and preserves positive credit for the weight (+1.681) and foldability (+0.720) reasoning spans. Full values are logged training tensors, GRPO is the captured response-level baseline, and GDPO is recomputed on the same raw-reward batch using the paper's 0.4/0.4/0.2 weights, with each method's normalization and clipping applied.

This qualitative pattern is also supported quantitatively across the dataset: in our offline replay of 8,992 reasoning sentences (800 responses across 50 items), gating and sign-aware selection directly resolve 469 sentences with conflicting-sign candidates, and in Section 5.5 (Tables 5–6), AnchorCredit achieves 76.50% / 73.62% multilabel association micro-F1 (versus 61.09% / 64.43% for keyword matching and 38.05% / 48.74% for gradient attribution) and 92.29% primary-component precision after gating.

W2. Training cost, GPU hours, and inference cost.

Training time. We measured AnchorCredit’s overhead relative to GRPO on Qwen3.5-4B with 8×A100 GPUs:

MeasurementGRPOAnchorCreditChange
Steady-window time per step106.7 s144.1 s+35%
Complete training job22.71 h27.16 h+20%
Peak GPU memory per training process37.64 GiB37.73 GiB+0.09 GiB

The overhead comes from one no-gradient forward pass and the computation of sentence-level credit from attention. No additional backward pass is required. By comparison, RTT requires separately training a Qwen3.5-4B relevance discriminator and repeatedly invoking it during policy training. AnchorCredit uses the policy model’s own attention and requires no additional model.

Because the +35% steady-window step difference above includes policy differences in generated rollout lengths during training, we separately timed the additional no-gradient attention forward inside each AnchorCredit update on fixed responses to isolate its direct cost. Each length bin comprised three fixed-response runs with 50 measured updates each on eight A100 GPUs:

Fixed-input lengthExtra forward (s)Full update wall time (s)Forward share
Short1.74814.34912.18%
Medium2.16716.69012.98%
Long2.72821.33012.79%

Across 450 measured updates, total forward time was 12.69% of total update time. The numerator uses the maximum measured model-forward time across participating workers. The denominator starts at advantage computation and ends after the actor update and synchronization; generation, verifier reward computation, and reference-model inference are timed separately. In the full 600-step training trajectory, the entire attribution prepass, including forward, routing, communication, and driver finalization, took 12,798.27 of 85,306.61 seconds of training-step wall time (15.00%).

GPU hours. Multiplying the recorded job wall time by eight A100 GPUs gives approximately 181.68 nominal GPU-hours for GRPO and 217.28 for AnchorCredit. These are job-clock estimates rather than measurements of device binding and release.

Inference cost. AnchorCredit changes only training-time credit assignment. The trained model has the same architecture as its GRPO counterpart and requires no additional inference computation. We compared the two trained models on the same L20 GPU at concurrency 16, using three decoding seeds and 200 items per seed:

MetricGRPOAnchorCredit
Mean request latency9.39 s6.14 s
Mean output length442 tokens263 tokens
Time to first token0.40 s0.40 s
Total time for 200 requests135.6 s173.6 s

The AnchorCredit model produces shorter outputs on average and has lower mean request latency. However, a few requests in each run reach the 8,192-token generation limit, increasing the total batch completion time. These measured differences reflect the models’ output-length distributions rather than an additional inference operation required by AnchorCredit.

Benefits under a common time budget. To assess whether the overhead is justified, we used the recorded 22.65-hour job clock to GRPO’s step-599 checkpoint as a shared budget. For each method, we selected the latest checkpoint saved within that budget, with checkpoints saved every 100 steps: step 599 for GRPO and step 400 for AnchorCredit, reached after 19.15 hours.

MethodCheckpointTraining timeWeight RMAE↓Dimension RMAE↓Fold F1↑Validity↑
GRPOStep 59922.65 h30.8622.4889.5599.88
AnchorCreditStep 40019.15 h26.1320.5990.0799.38

AnchorCredit’s step-400 checkpoint achieves weight and dimension RMAE approximately 4.7 and 1.9 points lower than GRPO’s final checkpoint, respectively. Thus, it achieves lower error with less training time in this comparison. At the final checkpoint (step 599), AnchorCredit reduces weight RMAE from 30.86 to 25.12 (an 18.6% relative error reduction) and dimension RMAE from 22.48 to 19.68 (a 12.5% relative reduction), and improves SGD Joint Goal Accuracy by 6.00 percentage points (a 24.3% relative error reduction; Table 2).

The measurement windows and complete results are provided in Appendix G.3 (Tables 22–25). We hope these additional analyses address your concerns and would be happy to clarify any remaining questions.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer,

We would like to briefly follow up on our response to your review. We would appreciate your feedback on whether our clarifications address your concerns and whether any important questions remain unresolved. We would be happy to provide further clarification.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Follow-up to W2 on training cost with FlashAttention-2

Our submitted implementation ran an additional policy forward pass to recover attention for credit assignment. We extended FlashAttention-2 to save the attention rows needed for candidate anchors during generation [1]. The table reports mean step and phase times for GRPO, the submitted AnchorCredit implementation, and the new implementation. Each run used Qwen3.5-4B on eight A100 GPUs and generated 256 responses per step.

MetricGRPOSubmitted AnchorCreditAnchorCredit + FlashAttention-2 export
Mean training step (s)157.3124.3164.0
Generation (s/step)59.619.557.3
Reference log probabilities and values (s/step)45.038.137.4
Post-rollout attribution (s/step)—19.38.6
Policy training (s/step)44.739.851.5
Output tokens/response542290463
Output tokens/s during generation2,3273,8052,069
Output tokens/s over the full step882598722
Generation seconds per 1,000 output tokens0.430.260.48
Full-step seconds per 1,000 output tokens1.131.671.38

The new implementation used zero additional full-model attribution passes during the measurement window. Post-rollout attribution took 8.6 s/step, down from 19.3 s/step in submitted AnchorCredit (−55.3%). Compared with GRPO, the new implementation took 4.3% longer per step and generated 14.6% fewer output tokens. Across all eight GPUs, full-step time per 1,000 output tokens was 1.38 s vs. 1.13 s for GRPO (+22.1%). We divide the total measured time by the total number of output tokens. Each run sampled its own responses, and this window measures a portion of training.

In a separate fixed-output test comparing two versions that both reuse rollout attention, moving row selection from CPU to GPU reduced generation time by a median of 5.56% across eight same-GPU pairs.

[1] AnchorCredit FlashAttention-2 Supplement (AnchorCredit_FlashAttention2_Supplement.zip). The anonymous archive contains the kernel patch, GPU selector, timing data, and a script to reproduce the reported summaries.

Reviewer_26MMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper propose AnchorCredit, a reinforcement algorithm that adaptively assigns credits to tokens when there exists multiple verifiable rewards. The method uses the attention weights of the structured output to determine the weight of each reward on a sentence. After that, the advantage is calculated by gating the irrelevant rewards by a heuristic function, and is further enhanced by another heuristics based on the signs of the selected rewards. Across 3 mulit-reward benchmarks, the method shows consistent gains over RL baselines.

Critical strengths
  • The method is well motivated by the need of reward assignment under the setting of multiple rewards.
  • The method is simple and does not require additional networks for assignments.
  • The evaluation across dense and MoE architectures is a valuable addition, providing evidence that the proposed credit-assignment method is applicable to different model scales and configurations.
  • The introduction of ProductAttr provides a useful benchmark for investigating multi-component credit assignment in multimodal reinforcement learning, particularly for structured outputs involving heterogeneous attributes.
Critical weaknesses
  • It's not convincing that the proposed advantage gating and selection are useful. Table 16 suggests that using simple averaging produces comparable results. It also lacks a baseline of weighted averaging using the existing attention weights.
  • It's unclear why the paper applies Whitening when Table 17 shows that it has no major benefits on the performance.
  • The incremental improvements measured by RMAE doesn't justify the additional forward pass or custom kernel required by this method.
  • The method is evaluated only on three benchmarks that has small amount of verifiable rewards, so it's unclear whether it can scale with larger amount of verifiable rewards and across a diverse domain.
  • There is no justification on averaging the weight over all attention. For example, earlier and later layers can serve different purposes and attend to different parts of the context. And models with different hybrid architecture can have different number of full attention layers. The paper only tests the method on the Qwen family model.
Questions for authors

N/A

5 comments

We thank the reviewer for recognizing the motivation, the design that requires no additional network, and the value of the dense, MoE, and ProductAttr evaluations.

Across our ablations (W1, W2, W5), we note one key point: our core contribution is output-anchored credit assignment (routing component-specific advantages to reasoning sentences via output-anchor attention), rather than any specific post-processing rule. Compared with response-level GRPO (30.86 / 22.48 / 89.55), every output-anchored variant (Mean, Hard Top-1, and NoPerAttrWhiten in Tables 16–17, and WA in Table 20) substantially improves all three metrics (25.12–26.91 / 19.45–20.02 / 91.31–92.99). This shows that the main gain comes from output-anchor attention routing itself and is robust to downstream combination and normalization details, while gating, sign-aware selection, and whitening further refine attribution precision and scale consistency.

W1. Gating, sign-aware selection, and weighted averaging.

We added the attention-weighted baseline suggested by the reviewer, Weighted-All (WA). WA uses AnchorCredit's output-anchor attention weights wijkw_{ij}^k, removes gating, and averages over all available components. WA and the full method (Full) start from the same initial model and use identical training hyperparameters. Both are evaluated on the same 800 ProductAttr items with three decoding seeds, yielding 4,800 predictions, with 100% valid outputs for both.

MethodWeight RMAE↓Dimension RMAE↓Fold F1↑
Full25.7619.5692.60
WA26.9119.9592.54

Full achieves lower Weight and Dimension RMAE (by 1.15 and 0.39 points) and slightly higher Fold F1 than WA. Beyond these endpoint scores, their credit-assignment behavior differs substantially. Routing statistics from the first training step show that WA routes every sentence, averaging positive and negative component advantages together for 67.63% of sentences. Full routes a sentence only when its attention association is sufficiently clear: 10.10% of sentences fall back to the response-level advantage because their associations are ambiguous, and only 5.25% have opposite-sign candidates. For these sentences, Full does not average; it adopts the advantage of the most strongly associated component. These percentages describe each run’s own first-step responses, rather than a fixed-response gate-on/gate-off comparison. Gating also improves attribution precision: in the offline association evaluation (Table 6), primary-component precision is 92.29% for AnchorCredit (gated) versus 84.28% for AnchorCredit (ungated), which—like WA—routes every sentence without gating. Gating and sign-aware selection therefore keep task performance while giving each reasoning sentence credit from clearly related components, rather than from a mixture of component advantages.

This is consistent with Mean and Full giving similar results in Table 16. The two rules differ only when gated candidates contain both positive and negative advantages. Such sentences account for only 5.25% of the first-step sample (604/11,502), so this diagnostic suggests that the distinction between the two rules is localized. We have added the WA results, routing statistics, and this discussion to Section 5.4 and Appendix G.1 (Table 20) of the revised paper.

W2. Motivation and role of pre-routing whitening (Table 17).

Following GDPO (Liu et al., 2026b), Eq. 1 first normalizes each reward component within each prompt group (Aik=NG(rik)A_i^k = \mathcal{N}_G(r_i^k)) to decouple heterogeneous reward scales. However, whereas GDPO immediately aggregates component advantages into a single response-level scalar and applies batch whitening only to that sum, AnchorCredit routes distinct component advantages to different sentences, averages multiple same-sign candidates (Eq. 7), compares candidate signs for conflict handling, and falls back to the response-level advantage AiA_i on unrouted sentences. Even after group normalization (Eq. 1), different components can exhibit unequal effective batch variances and slight mean shifts: for example, binary attributes like Foldability produce more zero-variance prompt groups (when all rollouts in a group predict the same binary label, yielding Aik=0A_i^k = 0) than continuous attributes (Weight and Dimensions). Pre-routing whitening (Eq. 2) is therefore motivated as a batch-level calibration step that places each component-advantage column A~ik\widetilde{A}_i^k and the fallback advantage A~i\widetilde{A}_i on a common zero-mean, unit-variance scale before routing, same-sign averaging, and sign comparisons.

In our offline replay of 8,992 reasoning sentences holding responses, attention, and raw advantages fixed, enabling pre-routing whitening increases mixed-sign cases—where gated candidates have opposite signs and the most strongly associated component is selected—from 278 to 469, reverses the credit sign for 93 sentences, and changes credit from zero to nonzero for 3,072 sentences, confirming that Eq. 2 actively calibrates cross-component sign comparisons and routing.

At the same time, we emphasize that pre-routing whitening is an auxiliary calibration step rather than a load-bearing component of our framework. As Table 17 shows, adding pre-routing whitening reduces Weight RMAE by 0.54 points over NoPerAttrWhiten (25.12 vs. 25.66) with comparable Dimension RMAE (19.68 vs. 19.45) and Fold F1 (91.31 vs. 92.32), while both variants outperform response-level GRPO (30.86 / 22.48 / 89.55) by a wide margin across all three metrics. This confirms that the primary gain comes from output-anchored credit assignment itself, which remains effective and robust without pre-routing whitening. We have clarified this design motivation, its relation to GDPO, and the offline replay analysis in Section 4.1, Appendix D.3, and Appendix G.1 of the revised paper.

W3. Computational cost and performance gains.

AnchorCredit requires no custom kernel, no additional network or trainable parameters, and no extra backward pass. Our current implementation uses a single standard no-gradient forward pass to extract anchor attention only because off-the-shelf FlashAttention does not return attention weights; because routing only requires attention from the few anchor tokens (∣Sik∣≪T|S_i^k| \ll T) rather than the full T×TT \times T matrix, computing or accumulating anchor–sentence attention on the fly during the existing policy forward pass (as noted in Section 4.3) is lightweight and would theoretically reduce attribution overhead to near zero.

To isolate the direct cost of this no-gradient forward pass from policy differences in generated rollout lengths, we timed it on fixed responses across 450 measured updates on 8×A100 GPUs: the extra forward accounts for only 12.69% of update wall time (excluding generation, reward evaluation, and reference-model inference; Appendix G.3, Table 22). In the full end-to-end training runs on Qwen3.5-4B (which include differences in generated lengths), the observed job-level costs are:

MeasurementGRPOAnchorCreditChange
Steady-window time per step106.7 s144.1 s+35%
Complete training job22.71 h27.16 h+20%
Peak GPU memory per training process37.64 GiB37.73 GiB+0.09 GiB

Even under the same wall-clock budget—using the recorded 22.65-hour job clock to GRPO’s final step-599 checkpoint—AnchorCredit's latest saved checkpoint within that budget (step 400, reached at 19.15 h) already achieves weight and dimension RMAE 4.74 and 1.88 points lower than GRPO’s final checkpoint (Appendix G.3, Table 24). At matched training steps (100/200/300/599), AnchorCredit outperforms GRPO in all eight weight and dimension comparisons, and at the final checkpoint reduces weight RMAE from 30.86 to 25.12 (18.6% relative error reduction) and dimension RMAE from 22.48 to 19.68 (12.5% relative reduction), with zero inference overhead (Table 25).

W4. More reward components and domain diversity.

In our submitted evaluations, while ProductAttr and Nutrition5k use K=3K=3 rewards, ProductAttr-FiveTask (Appendix C.2, Table 13) evaluates K=5K=5 components (improving four of them), and SGD (Table 2) involves K=2–13K=2\text{--}13 requested slots per dialogue frame (mean 8.36), where each slot is scored as a separate reward component and 97.31% of frames have K>3K>3 (+6.00 JGA points over GRPO).

To further examine scaling with KK and domain diversity, we added two analyses (Appendix G.4, Tables 26–27):

  1. Stratification by component count KK on SGD (Table 27). Grouping all 46,116 SGD test frames (4,201 dialogues) by the number of requested slots KK shows substantially larger JGA gains when more components are present:
Decoding seedK>3K>3 (97.31% of frames)K≤3K\le 3Difference
42+6.00+2.11+3.88
43+6.13+2.59+3.54

This confirms that AnchorCredit's benefit grows as cross-component credit misassignment becomes more severe at larger KK. 2. New domain evaluation on RAMS (Ebner et al., ACL 2020; Table 26). Beyond product attributes (ProductAttr), visual nutrition regression (Nutrition5k), and dialogue tracking (SGD), we added RAMS, a public news-domain event argument extraction benchmark (871 test documents, 1–5 event roles per document, 57.29% with K>3K>3). AnchorCredit improves Link F1 from 43.87 to 44.31 (+0.45 points), per-role span F1 from 58.35 to 58.82 (+0.47 points), and document-level joint accuracy from 16.30% to 17.95% (+1.65 points, or 10.1% relatively). Gains on RAMS are positive but more modest than on ProductAttr or SGD because locating text spans in news passages relies less on multi-step reasoning chains.

Reference: Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. Multi-Sentence Argument Linking. In Proceedings of ACL 2020, pp. 8057–8077.

W5. Layer aggregation and model families.

We use all-layer averaging to avoid model-specific layer tuning: the implementation averages over all full-attention layers exposing explicit attention matrices, while linear-attention layers in hybrid architectures are automatically excluded. For Qwen3.5-4B, these are the eight full-attention layers (indices 3, 7, …, 31).

In this rebuttal, we tested how layer selection affects sentence–component association ranking by replaying 1,024 fixed GPT responses from 128 products (7,807 reasoning sentences) through frozen Qwen3.5-4B, varying only the aggregated layers and scoring whether the top-attention component belongs to the GPT-5.6 Terra reference-label set (counting empty-reference sentences as misses; Appendix G.2, Table 21(a)):

Layer group (zero-based indices)Top-1 agreement (%)
All (3, 7, 11, 15, 19, 23, 27, 31)94.26%
Early (3, 7, 11)93.06%
Middle (15, 19, 23)91.00%
Late (27, 31)83.99%

All-layer averaging achieves the highest Top-1 agreement, exceeding the best subset (Early) by 1.20 percentage points (95% paired product-bootstrap interval [+0.77, +1.66]), supporting all-layer averaging as a parameter-free default.

To test the same full-attention aggregation rule on a different hybrid architecture, we replayed the same frozen responses through Gemma4 E4B. Its seven global-attention layers (indices 5, 11, 17, 23, 29, 35, 41) give 93.39% Top-1 agreement when averaged together, versus 81.54%, 91.79%, and 87.69% for the early middle , and late subsets. The gain over the best subset is 1.60 percentage points (95% paired product-bootstrap interval [+0.87, +2.28]; Appendix G.2, Table 21(b)). Both models average over their respective number of eligible layers rather than summing their scores. Thus, despite different layer counts and hybrid architectures, the same aggregation rule outperforms the tested depth subsets on this association diagnostic. This does not assume that early and late layers serve identical functions, or establish that equal averaging is optimal among all possible weighting schemes; subset depth and layer count also vary together. The appendix separately reports averaging all 42 explicit-attention layers, including local layers (93.77%), as an additional diagnostic. These frozen-response results concern association ranking, not downstream gains from Gemma RL training.

Regarding model families, the submitted paper already includes an evaluation beyond Qwen on Gemma3-4B-IT in Appendix D.1 (Table 14). Unlike Qwen3.5, Gemma3 alternates local sliding-window attention and global attention; applying the same parameter-free layer-averaging rule reduces dimension RMAE from 133.86 to 50.88 and improves Fold F1 from 55.91 to 59.15, with a slight increase in weight RMAE (51.53 to 52.25). We have highlighted this cross-family result more prominently in the main text (Section 5.3) and added the layer-group analysis to Appendix G.2 (Table 21) of the revised paper.

Reviewer_ULdbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the cross‑component credit misassignment problem, where errors in some output components of a response incorrectly penalize other well‑performing components together with their associated reasoning segments. It proposes a plug‑in output‑anchored credit assignment algorithm that requires no additional human‑annotated relevance labels nor a separate relevance discriminator. The effectiveness of the proposed approach is validated on both the authors’ newly‑constructed dataset and several public benchmark datasets.

Critical strengths

(1) Proposing AnchorCredit, a plug‑in module that leverages output anchors and the policy model’s native attention to perform sentence‑component attribution and assign component‑granularity credit. It requires no additional relevance annotations during training and can be integrated with GRPO, DAPO and GDPO. (2) Constructing ProductAttr, a multi‑attribute image‑text benchmark dataset consisting of 7,200 real product samples.

Critical weaknesses

Though the effectiveness of the proposed method is demonstrated across multiple benchmarks, all main experiments are confined to the Qwen model family. Even the evaluated Qwen‑35B‑A3B MoE model still does not reach the mainstream hundred‑billion‑parameter scale. The paper lacks validation of the method on both alternative model families and mainstream hundred‑B‑parameter large‑scale models. Only limited experiments are carried out on a small‑sized Gemma3-4B-IT from a different model family.

Questions for authors

The self‑constructed ProductAttr dataset appears reasonably designed. Aside from Nutrition5k and SGD that are already used in the paper, could the authors share experimental results on more authoritative public benchmarks to further verify the generalizability of AnchorCredit?

3 comments

We thank the reviewer for recognizing the contribution of output-anchored credit assignment and ProductAttr. We address the model-family and scale question and the request for additional public benchmarks below.

W1. Model families and model scales.

Our tasks require native vision–language understanding combined with multi-step reasoning and strict JSON schema adherence. In the current open-weight ecosystem at trainable RL scales, Qwen and Gemma are the two primary model families that natively support multimodal inputs alongside strong reasoning and instruction following, whereas most other open-weight families are either text-only or lack native vision–language reasoning backbones. Importantly, Qwen3.5 and Gemma also represent distinct hybrid attention topologies: Qwen3.5 interleaves linear attention (Gated DeltaNet) with full attention, whereas Gemma alternates local sliding-window attention with global attention. Applying AnchorCredit’s parameter-free full-attention averaging across both architectures without model-specific tuning confirms that our routing mechanism does not rely on Qwen-specific attention patterns (on Gemma3-4B-IT in Table 14, dimension RMAE decreases from 133.86 to 50.88, a reduction of 82.98 percentage points, and Fold F1 improves by 3.24 points).

Regarding hundred-billion-parameter (100B+) models, online RLVR requires generating multiple long reasoning rollouts per prompt at every training step (256 rollouts of up to 8,192 tokens per step in our setup) and updating policy weights online, which makes 100B+ multimodal RL training prohibitively expensive for academic compute budgets. Consequently, our evaluated scales (4B dense, 9B dense, and 35B-A3B MoE) match—and at 35B exceed—the standard experimental regime across foundational and recent RLVR and credit-assignment studies, including DeepSeekMath/GRPO (Shao et al., 2024; 7B), DAPO (Yu et al., 2025; up to 32B), GDPO (Liu et al., 2026b; 1.5B–7B), GD2PO\text{GD}^2\text{PO} (Liu et al., 2026a; 3B–8B), RTT (Xu et al., 2026; 4B–8B), VinePPO (Kazemnejad et al., 2025; 7B), and FlowTracer (Dong et al., 2026; 7B–32B). Within our evaluated range, gains do not diminish with scale: the largest weight RMAE reduction (−6.93 points) is achieved on the largest 35B-A3B model (Table 3b). Moreover, cross-component credit misassignment (AiAik<0A_i A_i^k < 0) arises from scalar-reward aggregation in the RL objective itself rather than small model capacity—even in GPT-5 responses (Section 5.5), component-specific reasoning and cross-attribute trade-offs remain prominent. Because AnchorCredit adds no trainable parameters or auxiliary scale-matched discriminator (unlike RTT), it can be applied directly to 100B+ models when cluster resources permit. We have clarified both points in Section 5.3, Section 6, and Appendix D.1 of the revised paper.

Q1. Additional public benchmarks.

Existing public benchmarks generally fall into two categories: reasoning-intensive RLVR suites (e.g., GSM8K, MATH, LiveCodeBench), which require multi-step derivation but provide only a single scalar outcome reward, or traditional multi-field extraction datasets (e.g., MAVE; Yang et al., 2021), which contain multiple verifiable fields but rely primarily on surface-level text extraction rather than multi-step reasoning. Because very few public benchmarks jointly require substantive intermediate reasoning and multiple separately verifiable outputs, we constructed and will release ProductAttr specifically to fill this gap.

To nonetheless address your question on an additional authoritative public benchmark beyond Nutrition5k and SGD, we evaluated AnchorCredit on RAMS (Ebner et al., ACL 2020), a classic benchmark for document-level event argument extraction. Given a news document, the model extracts arguments for 1–5 requested event roles, each scored separately across all 871 test documents and 2,023 gold argument links (three decoding seeds; corpus-level Link F1 verified against the official scorer). Note that RAMS is fundamentally an extractive task—locating text spans in a passage—and relies much less on multi-step intermediate reasoning than physical-attribute or nutritional estimation, making it a less direct fit for reasoning-level credit assignment:

MetricGRPOAnchorCredit
Link F143.8744.31
Per-role span F158.3558.82
Joint accuracy16.3017.95
Format validity99.8599.46

Joint accuracy requires all requested role arguments in a document to be extracted correctly. AnchorCredit achieves higher Link F1 (+0.45 points), per-role span F1 (+0.47 points), and document-level joint accuracy (+1.65 points, from 16.30% to 17.95%), with slightly lower format validity (Appendix G.4, Table 26; Table 27 also adds a per-slot-count KK breakdown on SGD). Consistent with our analysis above, the gains on RAMS are positive but modest precisely because span extraction depends less on multi-step reasoning chains, whereas on tasks where intermediate reasoning directly drives multi-component predictions (ProductAttr, Nutrition5k, and SGD), routing component-specific credit yields much larger improvements.

Reference: Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. Multi-Sentence Argument Linking. Proceedings of ACL 2020, pp. 8057–8077.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer,

We would like to briefly follow up on our response to your review. We would appreciate your feedback on whether our clarifications address your concerns and whether any important questions remain unresolved. We would be happy to provide further clarification.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.