acceptodds
Under review as a conference paper at ICLR 2027

Answer Is Not Enough: Causal Mediation for Reasoning Credit Assignment in GRPO

Abstract

Group Relative Policy Optimization (GRPO) optimizes reasoning policies using verifiable final answer rewards. However, final answer correctness alone may not provide an adequate learning signal for the reasoning process. Our preliminary investigation shows that, when the answer condition is fixed, changing the reasoning process leads to an accuracy difference of up to 11.8%, indicating that the reasoning process provides information beyond the final answer. Motivated by this observation, we propose CaMeL (Causal Mediation Lens), a reasoning credit assignment method based on causal mediation that models the reasoning process as a mediator to estimate its contribution to the answer decision. CaMeL estimates this contribution with two signals. Reasoning Causal Effect (RCE) measures the effect of the reasoning process on the correct answer probability relative to direct answering, capturing the direction and magnitude of its contribution. Reasoning Causal Specificity (RCS) measures whether the change in the answer distribution induced by the reasoning process exceeds those induced by shuffled and irrelevant reasoning processes, retaining only contributions beyond the control reference. The two signals form a mediator contribution, which is incorporated into rollout scoring by adjusting the Reason credit without altering the final answer reward. Experiments show that CaMeL outperforms Vanilla GRPO on logical reasoning benchmarks and generalizes to out-of-distribution (OOD) settings. Quantitative analyses and human evaluation further show that the mediator contribution aligns well with human judgments of the reasoning process, with over 90% directional agreement among activated signals.

Primary area:
generative models
Keywords:
Large Language Model Post-Training, Reinforcement Learning with Verifiable Rewards, Group Relative Policy Optimization, Reasoning Credit Assignment, Causal Mediation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3oRCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CaMeL uses a frozen scorer to assess how a generated reasoning trace changes support for the correct answer. Shuffled and irrelevant traces act as controls, and the resulting signal adjusts reasoning-span advantages in GRPO. The evaluation covers three backbones, with experiments on logical reasoning, OOD tasks, and event reasoning.

Critical strengths

S1. Final-answer rewards can assign identical credit to reasoning traces of different quality. The preliminary experiments make this limitation clear and give a concrete motivation for the method.

S2. The gains extend across models and tasks. Table 3 reports 64.20% average accuracy for CaMeL, versus 61.98% for GRPO and 62.27% for OAR-G, and average OOD accuracy improves under both source-task settings.

S3. In the blinded human evaluation, RCE distinguishes supportive from contradictory reasoning much better than the GRPO advantage. This is useful evidence for the signal, while the component ablations help identify which parts of the method contribute to the gains.

Critical weaknesses

W1. I could not identify a final evaluation split separate from checkpoint selection. Appendix A.2 divides the development sets into calibration and checkpoint-selection partitions, but Appendix A.3 does not specify a separate partition for the final results in Table 3. All reported accuracies are consistent with integer counts over the selection partitions: 300 ReClor examples and 942 LogiQA2.0 examples. That alone does not prove data reuse. It does, however, leave a specific concern about checkpoint-selection bias that the evaluation description needs to resolve.

W2. The controls differ from the original trace in ways that could affect the scorer independently of reasoning quality. Appendix A.2 removes explicit answer-revealing expressions from the shuffled control without saying whether the original trace receives the same treatment. The irrelevant control is a short, fixed statement that explicitly says it is unrelated. A positive specificity score could therefore reflect answer cues or differences in context, so these controls do not cleanly isolate reasoning structure.

W3. Removing RCS changes average accuracy only from 64.20% to 64.09% in Table 3. The agreement above 90% is also conditional on signal activation, which covers 48.3% and 55.3% of decision-relevant examples. I would like to see a comparison with a filter based on |Δp| at the same coverage. That would help determine whether the intervention controls add anything beyond simple signal filtering.

W4. Several component gains are small, and the main comparisons include neither repeated training seeds nor confidence intervals. Without those results, I cannot tell whether the differences are stable across training runs.

Questions for authors

Q1. Which exact partitions produced Table 3? Were any of these examples also used for checkpoint selection?

Q2. Are answer-revealing expressions removed from both the original and shuffled traces? How do the results change when controls are matched for length and answer content?

Q3. At the same coverage, how does RCS compare with a filter based on |Δp| for human agreement and downstream accuracy?

Q4. Do the gains over the strongest baseline and the RCS ablation persist across training seeds?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3oRC (Part 1/2)

Summary

Thank you for your review and suggestions. To address the concerns, we conducted additional evaluations and analyses covering evaluation reliability, control specificity, and the contribution of RCS. The main results are presented below.

Evaluation Protocol and Supplementary Multi-Seed Results (W1, W4; Q1, Q4)

The test results for ReClor are unavailable because its official test server is closed. Table 3 in the original submission reports accuracy on the full development sets, including the checkpoint-selection subsets. We have kept Table 3 unchanged and conducted supplementary evaluations to examine whether the observed gains persist on evaluation data not used for checkpoint selection.

For ReClor, under the revised evaluation protocol, we use the same 200 development examples for confidence-threshold calibration and checkpoint selection, reserving the remaining 300 examples as an evaluation set. Any examples overlapping with this evaluation set are removed from the training data. For LogiQA2.0, we retain the original 627/942 calibration/selection split and additionally evaluate on the test set. Thus, the results in Table A provide additional evidence beyond the original development-set evaluation.

Due to resource and time constraints, we conducted experiments using Qwen3-4B-Instruct-2507 with three random seeds (42, 43, and 44) on ReClor and LogiQA2.0. The in-domain and OOD results are reported in Tables A and B, respectively. We also report mean accuracy with 95% confidence intervals, calculated across the three seed-level results using the Student’s t-distribution.

Table A. Accuracy on ReClor and LogiQA2.0 (Three Random Seeds: 42, 43, and 44; 95% CI)

MethodReClorLogiQA2.0
Base75.11 [74.64, 75.58]62.32 [61.87, 62.77]
SFT77.00 [68.85, 85.15]66.39 [57.05, 75.73]
Vanilla GRPO79.22 [78.75, 79.69]64.89 [59.52, 70.26]
OAR-G78.56 [75.11, 82.01]65.90 [62.99, 68.81]
GRPO-λ80.11 [76.19, 84.03]66.06 [63.72, 68.40]
CaMeL w/o Mediation79.56 [77.47, 81.65]66.84 [65.20, 68.48]
CaMeL w/o RCS79.89 [78.95, 80.83]66.97 [65.18, 68.76]
CaMeL w/o Asym. Integr.80.00 [77.62, 82.38]67.63 [65.69, 69.57]
CaMeL80.33 [79.51, 81.15]67.76 [65.90, 69.62]

Table B. OOD Generalization Performance (Three Random Seeds: 42, 43, and 44; 95% CI)

Source TaskMethodSocialIQAα-NLI
ReClorBase66.77 [66.57, 66.97]71.43 [70.78, 72.08]
Vanilla GRPO68.83 [65.48, 72.18]72.76 [70.72, 74.80]
OAR-G68.75 [68.23, 69.27]72.80 [72.23, 73.37]
GRPO-λ70.54 [69.40, 71.68]73.22 [72.97, 73.47]
CaMeL70.57 [69.20, 71.94]74.17 [72.70, 75.64]
LogiQA2.0Base66.77 [66.57, 66.97]71.43 [70.78, 72.08]
Vanilla GRPO67.83 [64.63, 71.03]72.13 [71.06, 73.20]
OAR-G68.82 [66.39, 71.25]72.24 [70.85, 73.63]
GRPO-λ68.82 [67.68, 69.96]72.50 [72.10, 72.90]
CaMeL70.93 [68.79, 73.07]73.46 [72.57, 74.35]

We find that CaMeL consistently outperforms other methods on both datasets, indicating that the observed improvements are not merely an artifact of using checkpoint-selection data for final evaluation.

To further examine the consistency of CaMeL’s gains, Table C reports per-seed and mean accuracy differences between CaMeL and GRPO-λ, as well as between CaMeL and CaMeL w/o RCS, on ReClor and LogiQA2.0.

Table C. Paired Seed-Level Accuracy Differences (Three Random Seeds: 42, 43, and 44)

ComparisonDatasetSeed 42Seed 43Seed 44Mean Δ (pp)
CaMeL vs. GRPO-λReClor-1.33-0.33+2.33+0.22
LogiQA2.0+2.29+2.13+0.69+1.70
CaMeL vs. w/o RCSReClor-0.33+0.67+1.00+0.44
LogiQA2.0+1.37+0.990.00+0.79
  1. Comparison with the strongest baseline. Table A shows that CaMeL achieves higher mean in-domain accuracy than GRPO-λ on both datasets. Table B further shows higher mean OOD accuracy in all four evaluations. As shown in Table C, the gains are consistent across all seeds on LogiQA2.0.
  2. Contribution of RCS. Table A shows that CaMeL improves mean accuracy over CaMeL w/o RCS on ReClor and LogiQA2.0. Table C further shows positive gains in two out of three seeds on each dataset.

Overall, these results provide additional evidence supporting CaMeL’s performance gains and the contribution of RCS. We will incorporate these results into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 3oRC (Part 2/2)

Answer-Revealing and Matched Controls (W2; Q2)

We remove answer-revealing expressions from both the original and shuffled reasoning traces using the same procedure before computing their scores. We have clarified this in Appendix A.2 and added a sensitivity analysis with alternative irrelevant controls in Appendix C.2.

To check whether RCS depends on the original irrelevant control, we conduct an additional experiment with three alternative controls. These controls consist of reasoning texts from unrelated questions. Each alternative is matched to the original reasoning in token length (within ±10%). We also remove any answer-revealing expressions from these texts. We evaluate RCS coverage, directional agreement, and incorrect-credit weight share for each alternative control. The average results are shown in Table D.

Table D. Sensitivity to Length-Matched Irrelevant Controls (Mean over 3 Variants)

DatasetControlCoverage (%)Dir. Agree. (%) ↑Incorrect Credit Weight Share (%) ↓
ReClorOriginal50.0090.701.81
Matched (Mean)57.0093.921.59
LogiQA2.0Original54.0094.120.17
Matched (Mean)58.3393.010.16

The results show that RCS maintains high directional agreement (above 90%) and low incorrect-credit weight shares across different irrelevant controls. This suggests that RCS does not rely solely on the original short irrelevant control. We have added this analysis to Appendix C.2.

RCS versus Magnitude Filtering (W3; Q3)

RCS uses shuffled and irrelevant controls to filter and reweight reasoning contributions. To examine whether RCS provides benefits beyond simple filtering based on |Δp|, we conduct two additional experiments: a matched-coverage human evaluation (Appendix C.3) and a downstream accuracy comparison (Appendix B.3).

  1. Human Evaluation at Matched Coverage. We compare RCS-weighted RCE with |Δp|-based filtering using the human-annotated samples from Table 6. The |Δp|-based method selects the top-k examples with the largest |Δp|. We set k to the number of nonzero RCS signals and preserve the sign of Δp. Thus, both methods have the same coverage.

    We evaluate the two methods using Directional Agreement and Incorrect Credit Weight Share. Directional Agreement measures the proportion of activated Supports/Contradicts examples whose credit directions agree with human annotations. Incorrect Credit Weight Share measures the proportion of total credit weight assigned to examples with incorrect credit directions. The results are shown in Table E.

    Table E. Human Evaluation at Matched Coverage

    DatasetMethodCoverageDir. Agree. ↑ (95% CI)Incorrect Credit Weight Share ↓ (95% CI)
    ReClor|Δp|-based filtering48/10090.70% [80.39, 100.00]7.98% [0.00, 16.19]
    RCS48/10090.48% [80.43, 97.87]1.68% [0.02, 4.36]
    LogiQA2.0|Δp|-based filtering55/10096.15% [86.79, 100.00]1.96% [0.00, 6.07]
    RCS55/10092.31% [84.00, 98.21]0.09% [<0.001, 0.31]

    The results show that RCS achieves directional agreement comparable to that of |Δp|-based filtering, while substantially reducing the Incorrect Credit Weight Share on both datasets. This suggests that RCS mainly contributes through credit reweighting rather than improving directional signal selection.

  2. Downstream Accuracy. To examine whether this difference in credit assignment affects downstream performance, we compare the original CaMeL with a variant that replaces RCS with matched-coverage |Δp|-based filtering. Both variants are trained using Qwen3-4B-Instruct-2507 under identical training settings. We evaluate them on ReClor and LogiQA2.0. The results are shown in Table F.

    Table F. Downstream Accuracy Comparison (Seed 43)

    DatasetMethodAccuracy (%)95% CI
    ReClorCaMeL (|Δp|-based filtering)78.33[73.33, 82.62]
    CaMeL79.33[74.39, 83.53]
    LogiQA2.0CaMeL (|Δp|-based filtering)66.46[63.86, 68.97]
    CaMeL67.15[64.56, 69.64]

    CaMeL achieves higher accuracy on both datasets. These results provide additional evidence that RCS offers benefits beyond |Δp|-based filtering.

Overall, the additional experiments provide further evidence of CaMeL’s effectiveness, demonstrating its performance gains and the contributions of its key components. We have also addressed the reviewers’ concerns through additional evaluations and analyses, and incorporated the relevant clarifications and selected results into the revised manuscript.

Thank you for your response. The explanations cleared most of my questions, especially about the sources of performance gains and the robustness of the method. Therefore, I have raised my score to 3.

AuthorsOpenReview ↗

Thank You for Your Consideration

Dear Reviewer,

We appreciate the time and effort you have devoted to reviewing our work. We are glad that our responses have clarified most of the concerns you raised. Should any questions or concerns remain, please feel free to let us know. We would be happy to address them.

Best regards,

The Authors

Reviewer_xw9UOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the tension often seen with reasoning models: we have an intuition that reasoning should lead to improvements, but does not always do this. and often the reasoning chains seem transparently not to be a real stepping stone to better answers. This paper proposes to help address this by introducing two new signals -- Reasoning Causal Effect (RCE) and Reasoning Causal Specificity (RCS) -- and seeking to show that they can contribute to improved reasoning and better task performance when used as reward signals with GRPO. The primary results are for Qwen and Llama models, and they show small improvements on in-domain and out-of-domain tasks.

Critical strengths

I certainly think it is important to be looking critically at model reasoning chains and thinking about how to ensure that they are richer and lead to improved test performance more reliably.

RCE and RCS seem like they are important aspects of a causal analysis of reasoning, and it seems like they are useful as reward signals.

Critical weaknesses

I'm open-minded about this paper, and I do see value in the technical ideas, even if the improvements seem small. However, there are lots of points of strong overlap with prior literature, enough to make me wonder whether this contribution is sufficiently original to warrant publication:

  1. The paper is missing some key citations. RCS is, in its core measurement, a special case of Causal Importance of Reasoning from Yu et al. 2026 (https://arxiv.org/abs/2604.22074), which is similar to TRACE (Wang et al. 2025; https://arxiv.org/abs/2310.06762). See also the ideas in Lanham et al. 2023 (https://arxiv.org/abs/2307.13702), which is cited by the connection at the level of the metric is not made clear. These papers go beyond this paper by measuring the causal effect of each token in the reasoning chain, with no reasoning vs. full reasoning them just extremes of that entire measurement.

  2. Yu et al. 2026 has many other points of similarity with this paper. While Yu et al. are largely focused on using their metrics as analysis tools, they do augment RL policies to see whether these metrics, as reward components, lead to improved performance. Yu et al. also report (in their section 2.3) what is in effect the same finding as in section 2.3 of the current paper (reasoning hurts for high-confidence examples).

  3. Chia et al. 2023 (https://arxiv.org/abs/2311.09277) used the framework of Wang et al. 2022 (https://arxiv.org/abs/2212.10001) to analyze CoT. Both use a metric with strong connections to RCS, but neither work is cited.

  4. I don't really see a sense in which it is justified to say that the paper involves a causal mediation analysis. That would imply an analysis of the total effect of reasoning on the answer. The DAG analysis (Paul et al. 2024) is an example of such an analysis. For the current paper, a lot of the necessary ingredients seem to be missing, in particular, a clear statement of a treatment contrast and a measured direct effect.

Questions for authors

Please see "Critical Weaknesses".

3 comments
AuthorsOpenReview ↗

Response to Reviewer xw9U

1. Summary

Thank you for your comments. We believe that CaMeL differs from the cited studies in research objectives and methods, yet complements them. We have reviewed these studies and will clarify how they relate to our work. Citations have been added to the Related Work section. The following sections detail these comparisons. We also clarify two reference mismatches:

  1. The address of Wang et al. (2025) (arXiv:2310.06762) refers to a continual learning benchmark, which is unrelated to our work. We suppose the intended paper may be Wang et al. (2025, https://arxiv.org/abs/2510.01367), which we discuss below.
  2. We could not find this finding in Section 2.3 of Yu et al. (2026). We believe you may be referring to Section 5.2, which we discuss below.

2. Comparison with Related Work

Differences from prior work are:

  1. Measurement. Yu et al. (2026) use CIR to measure how truncated reasoning changes the probability of the generated answer, without distinguishing support for or contradiction of the correct answer. CaMeL uses RCE to measure this distinction and RCS to filter changes that do not exceed the control conditions.
  2. Objective and Method. Wang et al. (2025) aim to detect implicit reward hacking by progressively truncating reasoning traces. CaMeL instead estimates reasoning contributions for GRPO credit assignment by comparing a frozen scorer's predictions under original reasoning, no-reasoning, and placebo reasoning conditions.
  3. Intervention Design. Lanham et al. (2023) assess CoT faithfulness by truncating, introducing errors into, or paraphrasing reasoning. CaMeL shuffles reasoning to disrupt logical order and irrelevant reasoning to remove task relevance, retaining only effects exceeding both controls.
  4. Intervention Target. Wang et al. (2022) investigate what makes few-shot CoT prompting effective by modifying the reasoning in demonstration examples. CaMeL, however, intervenes on the model-generated reasoning for the question, using no-reasoning, shuffled, and irrelevant reasoning conditions to estimate its contribution to answer predictions.
  5. Comparison Setting. Chia et al. (2023) use valid and invalid reasoning demonstrations to improve reasoning through contrastive prompting. CaMeL does not contrast correct and incorrect reasoning examples, but evaluates the contribution of generated reasoning relative to a no-reasoning baseline.

Overall, these studies and CaMeL address different aspects of reasoning and are therefore complementary.

We clarify the differences between Yu et al.'s Section 5.2 and our Section 2.3:

  1. Objectives. Yu et al. examine whether RLVR without explicit reasoning achieves similar gains on certain tasks. CaMeL examines how reasoning gains vary with direct-answer confidence to guide routing.
  2. Experimental Settings. Yu et al. compare RLVR training with and without reasoning across tasks characterized by CIR/SR. We group examples within the same task by direct-answer confidence, compare accuracy with and without reasoning, and determine the routing threshold.

3. RCS vs. CIR

We clarify the differences between RCS and CIR in two respects:

  1. Intervention design. CIR compares partial reasoning prefixes with complete reasoning, while RCS compares complete, shuffled, and irrelevant reasoning against a shared no-reasoning baseline.
  2. Measurement. CIR averages prediction differences for the generated answer across reasoning prefixes. In contrast, RCS measures distributional shifts over all candidate answers and subtracts the largest shift induced by the control conditions to reduce non-specific effects. Thus, CIR measures reasoning dependence, while RCS evaluates reasoning-specific distributional changes after placebo correction.

4. Use of Causal Mediation

Although CaMeL does not perform conventional direct and indirect effect decomposition, it draws on principles inspired by causal mediation analysis, particularly mediator structure and mediator intervention, as explained below:

  1. Mediator structure. Causal mediation analysis considers a mediator as an intermediate variable between treatment and outcome. Similarly, CaMeL treats reasoning as an intervenable mediator between the input and answer decision.
  2. Mediator intervention. CaMeL adopts the idea of mediator intervention by holding the question, candidate answers, and scorer fixed while varying the reasoning process. RCE measures the change in correct-answer probability, while RCS uses shuffled and irrelevant reasoning as placebo controls. Unlike Paul et al. (2024), we estimate reasoning contributions for the same question to improve credit assignment in GRPO, rather than decompose the total effect of changes in the question. We will clarify this distinction in the revised manuscript. Overall response. Overall, our work differs from the studies you mentioned in several aspects. We will clarify these differences in the revised manuscript.
Reviewer_xw9UOpenReview ↗

Useful context; raising my scores

I appreciate the authors' productive response (and my apologies for mixing up some of the citations). The response helps convince me that this is a useful new contribution to this space, so I will raise my overall score by 1.

AuthorsOpenReview ↗

Thank You for Your Time and Feedback

Dear Reviewer,

Thank you for your time and thoughtful feedback. We appreciate your comments and the opportunity to clarify our work. We hope our responses have addressed your concerns. Should you have any further questions, we would be happy to provide additional clarification.

Best regards,

The Authors

Reviewer_8bXNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CaMeL, a method considering reasoning process when using GRPO. It uses a frozen scorer to measure how the generated reasoning process changes the probability of the correct answer, then filters this signal using shuffled and irrelevant reasoning processes. The resulting correction is applied only to the reasoning span. The method also uses confidence-based routing to decide when to generate a reasoning process. The problem is well motivated, and the results across multiple backbones and tasks are encouraging. Reusing a frozen initialization model as the scorer is also practical.

Critical strengths

The paper is easy to follow, and the method is practical: it reuses the initial model as a frozen scorer. The evaluation covers three backbones, several reasoning tasks, and OOD settings. The human evaluation is also useful. The overall results are encouraging. In Table 3, CaMeL improves average accuracy from 61.98% with vanilla GRPO to 64.20%.

Critical weaknesses
  1. The source of the gains is unclear. CaMeL also changes confidence-based routing and advantage normalization. The version without mediation already improves average accuracy from 61.98% to 63.16%; the full method reaches 64.20%. Experiment results of baselines also with routing and normalization would be helpful.

  2. The causal claim needs more support. The method measures an intervention effect on a frozen scorer, which may differ from the evolving policy. Does this reliably reflect the policy’s reasoning contribution?

  3. RCS adds little on average. Removing it reduces average accuracy by only 0.11. Higher human agreement comes with much lower coverage. A comparison against simple RCE-magnitude filtering at matched coverage would be useful.

  4. Evaluation details need clarification. Please specify the evaluation splits and confirm their independence from checkpoint selection. Multiple seeds are also needed to judge the smaller gains.

Questions for authors
  1. What exact splits and sample counts were used for the final results in Table 3?
  2. How much improvement remains when GRPO uses the same routing and advantage normalization?
  3. Does RCS outperform magnitude-based RCE filtering at matched coverage?
  4. Are answer cues removed consistently from both original and shuffled reasoning processes?
  5. How stable are the gains across seeds, and what is the additional training cost?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 8bXN (Part 1/3)

Summary

Thank you for your review and constructive suggestions. In response to your concerns, we have conducted additional experiments to examine the sources of performance gains, the reliability of the frozen scorer, and the role of RCS. We provide detailed responses below. To avoid ambiguity, we clarify two points:

  1. RCE-magnitude filtering. Based on our review of related studies and your description, we interpret RCE-magnitude filtering as selecting the top-kk examples by ∣Δp∣|\Delta p|, where kk matches the number of nonzero RCS signals in the same sample pool, while preserving the sign of Δp\Delta p. We conduct the following comparisons accordingly.
  2. Answer cues. Based on your description and our implementation, we interpret answer cues as expressions that reveal the final answer (e.g., "the answer is B"), rather than the reasoning content itself.

Evaluation Protocol and Multi-Seed Stability (W4; Q1, Q5)

The test results for ReClor are unavailable because its official test server is closed. In the original submission, Table 3 reports accuracy on the full development sets, including the subsets used for checkpoint selection. We conducted supplementary evaluations to examine whether the performance gains persist under independent evaluation.

To enable independent evaluation, we adopted the following data splits. For ReClor, we re-partitioned the development set: we used 200 examples for confidence-threshold calibration and checkpoint selection and reserved the remaining 300 as a held-out evaluation set. For LogiQA2.0, we retained the original 627/942 calibration/selection split and additionally evaluated on the official test set (1,572 examples).

Due to resource constraints, we conducted the supplementary experiments using Qwen3-4B-Instruct-2507 across three random seeds (42, 43, and 44). We evaluated all methods and ablations, including the newly added Matched GRPO baseline (which uses the same confidence-based routing and advantage normalization as CaMeL but does not use mediation-based credit assignment), on the evaluation sets for ReClor and LogiQA2.0 (Table A). We also evaluated the main methods on SocialIQA and α-NLI to assess OOD generalization (Table B). Results are reported as mean ± standard deviation across the three seeds.

Table A. Accuracy on ReClor and LogiQA2.0 (Mean ± SD over Seeds 42, 43, and 44)

MethodReClorLogiQA2.0
Base75.11 ± 0.1962.32 ± 0.18
SFT77.00 ± 3.2866.39 ± 3.76
Vanilla GRPO79.22 ± 0.1964.89 ± 2.16
Matched GRPO79.22 ± 1.0765.45 ± 0.29
OAR-G78.56 ± 1.3965.90 ± 1.17
GRPO-λ80.11 ± 1.5866.06 ± 0.94
CaMeL w/o Mediation79.56 ± 0.8466.84 ± 0.66
CaMeL w/o RCS79.89 ± 0.3866.97 ± 0.72
CaMeL w/o Asym. Integr.80.00 ± 0.9667.63 ± 0.78
CaMeL80.33 ± 0.3367.76 ± 0.75

Table B. OOD Generalization Performance (Mean ± SD over Seeds 42, 43, and 44)

Source TaskMethodSocialIQAα-NLI
ReClorBase66.77 ± 0.0871.43 ± 0.26
Vanilla GRPO68.83 ± 1.3572.76 ± 0.82
Matched GRPO70.28 ± 0.5573.43 ± 0.78
OAR-G68.75 ± 0.2172.80 ± 0.23
GRPO-λ70.54 ± 0.4673.22 ± 0.10
CaMeL70.57 ± 0.5574.17 ± 0.59
LogiQA2.0Base66.77 ± 0.0871.43 ± 0.26
Vanilla GRPO67.83 ± 1.2972.13 ± 0.43
Matched GRPO68.65 ± 0.0672.78 ± 0.46
OAR-G68.82 ± 0.9872.24 ± 0.56
GRPO-λ68.82 ± 0.4672.50 ± 0.16
CaMeL70.93 ± 0.8673.46 ± 0.36

Overall, CaMeL achieves the highest mean accuracy on both evaluation datasets (Table A) and in all four OOD evaluations (Table B), demonstrating the effectiveness and generalization ability of the proposed method. Furthermore, all component ablations yield lower mean accuracy on the two evaluation datasets than the full method (Table A), indicating that each component contributes to CaMeL's overall performance.

AuthorsOpenReview ↗

Response to Reviewer 8bXN (Part 2/3)

Sources of Performance Gains (W1; Q2)

To isolate the contribution of mediation-based credit assignment from confidence-based routing and advantage normalization, we introduced a Matched GRPO baseline. It uses the same confidence-based routing and advantage normalization as CaMeL but retains standard GRPO credit assignment without mediation correction. Results are presented in Tables A and B, respectively.

  1. Routing and Advantage Normalization. Compared with Vanilla GRPO, Matched GRPO yields no change in mean accuracy on ReClor and a gain in mean accuracy on LogiQA2.0 (Table A). It also achieves higher mean accuracy in all four OOD evaluations (Table B). These results indicate that routing and advantage normalization jointly contribute to performance, although their effects vary across evaluation settings.
  2. Mediation-based Credit Assignment. Compared with Matched GRPO, CaMeL achieves gains of 1.11 pp on ReClor and 2.31 pp on LogiQA2.0 (Table A). It also achieves higher mean accuracy in all four OOD evaluations (Table B). These results support the contribution of mediation-based credit assignment beyond the effects of these two components.

Frozen-Scorer Reliability (W2)

To examine whether the frozen scorer reliably reflects the evolving policy's reasoning contribution, we conducted additional experiments using Qwen3-4B-Instruct-2507 on ReClor and LogiQA2.0. Specifically, we compared RCE estimates produced by the frozen scorer and policy checkpoints on identical question–reasoning pairs. We used Sign Agreement to measure consistency in the direction of the RCE estimates and Spearman Correlation to measure consistency in their relative rankings. The results are shown in Table C.

Table C. Frozen Scorer–Policy Consistency (Seeds 42, 43, and 44)

DatasetSign AgreementSpearman ρ
ReClor95.42% [91.90, 98.24]0.860 [0.774, 0.928]
LogiQA2.096.00% [94.12, 97.62]0.900 [0.858, 0.934]

95% confidence intervals are reported in brackets.

As shown in Table C, the frozen scorer achieves over 95% sign agreement and strong rank correlations with the evolving policy on both datasets. These results support the reliability of the frozen scorer as a proxy for estimating the direction and relative magnitude of reasoning contributions.

AuthorsOpenReview ↗

Response to Reviewer 8bXN (Part 3/3)

RCS versus Magnitude Filtering at Matched Coverage (W3; Q3)

RCS uses shuffled and irrelevant controls to identify reasoning contributions that are specific to the original reasoning. It does not simply select examples with large contribution magnitudes. To examine its benefits beyond magnitude-based filtering, we added two experiments: a matched-coverage human evaluation in Appendix C.3 and a downstream accuracy comparison in Appendix B.3.

  1. Human Evaluation at Matched Coverage. We used the 200 human-annotated reasoning traces reported in the main text. We compared RCS with magnitude-based filtering at the same coverage. The magnitude-based filter selects the top-kk examples by ∣Δp∣|\Delta p|, where kk equals the number of nonzero RCS signals in the same sample pool.

    We report three metrics: Coverage, Directional Agreement, and Incorrect Credit Weight Share. Coverage measures the proportion of selected examples among all annotated examples. Directional Agreement measures the proportion of activated Supports/Contradicts examples whose credit directions agree with human annotations. Incorrect Credit Weight Share measures the proportion of total credit weight assigned to incorrectly directed examples. The results are shown in Table D.

    Table D. RCS vs. Magnitude Filtering at Matched Coverage

    DatasetMethodCoverageDirectional Agreement ↑ (95% CI)Incorrect Credit Weight Share ↓ (95% CI)
    ReClorRCE-magnitude filtering48/10090.70% [80.39, 100.00]7.98% [0.00, 16.19]
    ReClorRCS48/10090.48% [80.43, 97.87]1.68% [0.02, 4.36]
    LogiQA2.0RCE-magnitude filtering55/10096.15% [86.79, 100.00]1.96% [0.00, 6.07]
    LogiQA2.0RCS55/10092.31% [84.00, 98.21]0.09% [<0.001, 0.31]

    RCS does not achieve higher Directional Agreement than magnitude-based filtering. However, it substantially reduces the Incorrect Credit Weight Share on both datasets. This suggests that RCS mainly contributes through credit reweighting rather than directional signal selection.

  2. Downstream Accuracy. We further examined whether the benefits of credit reweighting translate into downstream performance. We compared CaMeL with a variant that replaces RCS with ∣Δp∣|\Delta p|-based filtering at the same coverage. Both variants were trained using Qwen3-4B-Instruct-2507 (seed 43) under identical settings and evaluated on ReClor and LogiQA2.0. The results are shown in Table E.

    Table E. Downstream Accuracy Comparison (Seed 43)

    Training DatasetMethodAccuracy (%)95% CI
    ReClorCaMeL (RCE-magnitude filtering)78.33[73.33, 82.62]
    ReClorCaMeL79.33[74.39, 83.53]
    LogiQA2.0CaMeL (RCE-magnitude filtering)66.46[63.86, 68.97]
    LogiQA2.0CaMeL67.15[64.56, 69.64]

    These results show that RCS improves reasoning credit assignment by reducing the weight assigned to incorrectly directed contributions, leading to higher downstream accuracy on both datasets.

Answer-Cue Removal (Q4)

We apply the same preprocessing procedure to both original and shuffled reasoning traces and remove explicit answer-revealing expressions before scoring. We have clarified this procedure in Appendix A.2 and added a sensitivity analysis using alternative irrelevant controls in Appendix C.2.

Additional Training Cost (Q5)

To assess the computational overhead introduced by CaMeL, we compared its training time with Vanilla GRPO. We conducted the experiments using Qwen3-4B-Instruct-2507 on ReClor and LogiQA2.0 across three random seeds (42, 43, and 44). We measured cumulative training time over the final 200 retained iterations. Since all runs used a single GPU, GPU-hours equal wall-clock hours. Results are reported as mean ± standard deviation across the three seeds.

Table F. Training Cost Comparison with Vanilla GRPO (Hours, Mean ± SD)

DatasetMethodWall-clock / GPU-hours
ReClorVanilla GRPO2.081 ± 0.797
ReClorCaMeL2.611 ± 1.079
LogiQA2.0Vanilla GRPO2.044 ± 0.640
LogiQA2.0CaMeL2.115 ± 0.789

Based on paired per-seed measurements, CaMeL increases training time by 24.38% ± 4.85% on ReClor and 4.72% ± 22.59% on LogiQA2.0. These results show that CaMeL introduces additional training overhead.

Overall, the additional evaluations support CaMeL's effectiveness and clarify the contributions of its key components. They also provide further evidence for the reliability of the frozen scorer and the role of RCS in improving reasoning credit assignment. Thank you for your constructive suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.