- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CaMeL uses a frozen scorer to assess how a generated reasoning trace changes support for the correct answer. Shuffled and irrelevant traces act as controls, and the resulting signal adjusts reasoning-span advantages in GRPO. The evaluation covers three backbones, with experiments on logical reasoning, OOD tasks, and event reasoning.
Critical strengths
S1. Final-answer rewards can assign identical credit to reasoning traces of different quality. The preliminary experiments make this limitation clear and give a concrete motivation for the method.
S2. The gains extend across models and tasks. Table 3 reports 64.20% average accuracy for CaMeL, versus 61.98% for GRPO and 62.27% for OAR-G, and average OOD accuracy improves under both source-task settings.
S3. In the blinded human evaluation, RCE distinguishes supportive from contradictory reasoning much better than the GRPO advantage. This is useful evidence for the signal, while the component ablations help identify which parts of the method contribute to the gains.
Critical weaknesses
W1. I could not identify a final evaluation split separate from checkpoint selection. Appendix A.2 divides the development sets into calibration and checkpoint-selection partitions, but Appendix A.3 does not specify a separate partition for the final results in Table 3. All reported accuracies are consistent with integer counts over the selection partitions: 300 ReClor examples and 942 LogiQA2.0 examples. That alone does not prove data reuse. It does, however, leave a specific concern about checkpoint-selection bias that the evaluation description needs to resolve.
W2. The controls differ from the original trace in ways that could affect the scorer independently of reasoning quality. Appendix A.2 removes explicit answer-revealing expressions from the shuffled control without saying whether the original trace receives the same treatment. The irrelevant control is a short, fixed statement that explicitly says it is unrelated. A positive specificity score could therefore reflect answer cues or differences in context, so these controls do not cleanly isolate reasoning structure.
W3. Removing RCS changes average accuracy only from 64.20% to 64.09% in Table 3. The agreement above 90% is also conditional on signal activation, which covers 48.3% and 55.3% of decision-relevant examples. I would like to see a comparison with a filter based on |Δp| at the same coverage. That would help determine whether the intervention controls add anything beyond simple signal filtering.
W4. Several component gains are small, and the main comparisons include neither repeated training seeds nor confidence intervals. Without those results, I cannot tell whether the differences are stable across training runs.
Questions for authors
Q1. Which exact partitions produced Table 3? Were any of these examples also used for checkpoint selection?
Q2. Are answer-revealing expressions removed from both the original and shuffled traces? How do the results change when controls are matched for length and answer content?
Q3. At the same coverage, how does RCS compare with a filter based on |Δp| for human agreement and downstream accuracy?
Q4. Do the gains over the strongest baseline and the RCS ablation persist across training seeds?
4 comments
Response to Reviewer 3oRC (Part 2/2)
Answer-Revealing and Matched Controls (W2; Q2)
We remove answer-revealing expressions from both the original and shuffled reasoning traces using the same procedure before computing their scores. We have clarified this in Appendix A.2 and added a sensitivity analysis with alternative irrelevant controls in Appendix C.2.
To check whether RCS depends on the original irrelevant control, we conduct an additional experiment with three alternative controls. These controls consist of reasoning texts from unrelated questions. Each alternative is matched to the original reasoning in token length (within ±10%). We also remove any answer-revealing expressions from these texts. We evaluate RCS coverage, directional agreement, and incorrect-credit weight share for each alternative control. The average results are shown in Table D.
Table D. Sensitivity to Length-Matched Irrelevant Controls (Mean over 3 Variants)
| Dataset | Control | Coverage (%) | Dir. Agree. (%) ↑ | Incorrect Credit Weight Share (%) ↓ |
|---|---|---|---|---|
| ReClor | Original | 50.00 | 90.70 | 1.81 |
| Matched (Mean) | 57.00 | 93.92 | 1.59 | |
| LogiQA2.0 | Original | 54.00 | 94.12 | 0.17 |
| Matched (Mean) | 58.33 | 93.01 | 0.16 |
The results show that RCS maintains high directional agreement (above 90%) and low incorrect-credit weight shares across different irrelevant controls. This suggests that RCS does not rely solely on the original short irrelevant control. We have added this analysis to Appendix C.2.
RCS versus Magnitude Filtering (W3; Q3)
RCS uses shuffled and irrelevant controls to filter and reweight reasoning contributions. To examine whether RCS provides benefits beyond simple filtering based on |Δp|, we conduct two additional experiments: a matched-coverage human evaluation (Appendix C.3) and a downstream accuracy comparison (Appendix B.3).
-
Human Evaluation at Matched Coverage. We compare RCS-weighted RCE with |Δp|-based filtering using the human-annotated samples from Table 6. The |Δp|-based method selects the top-k examples with the largest |Δp|. We set k to the number of nonzero RCS signals and preserve the sign of Δp. Thus, both methods have the same coverage.
We evaluate the two methods using Directional Agreement and Incorrect Credit Weight Share. Directional Agreement measures the proportion of activated Supports/Contradicts examples whose credit directions agree with human annotations. Incorrect Credit Weight Share measures the proportion of total credit weight assigned to examples with incorrect credit directions. The results are shown in Table E.
Table E. Human Evaluation at Matched Coverage
Dataset Method Coverage Dir. Agree. ↑ (95% CI) Incorrect Credit Weight Share ↓ (95% CI) ReClor |Δp|-based filtering 48/100 90.70% [80.39, 100.00] 7.98% [0.00, 16.19] RCS 48/100 90.48% [80.43, 97.87] 1.68% [0.02, 4.36] LogiQA2.0 |Δp|-based filtering 55/100 96.15% [86.79, 100.00] 1.96% [0.00, 6.07] RCS 55/100 92.31% [84.00, 98.21] 0.09% [<0.001, 0.31] The results show that RCS achieves directional agreement comparable to that of |Δp|-based filtering, while substantially reducing the Incorrect Credit Weight Share on both datasets. This suggests that RCS mainly contributes through credit reweighting rather than improving directional signal selection.
-
Downstream Accuracy. To examine whether this difference in credit assignment affects downstream performance, we compare the original CaMeL with a variant that replaces RCS with matched-coverage |Δp|-based filtering. Both variants are trained using Qwen3-4B-Instruct-2507 under identical training settings. We evaluate them on ReClor and LogiQA2.0. The results are shown in Table F.
Table F. Downstream Accuracy Comparison (Seed 43)
Dataset Method Accuracy (%) 95% CI ReClor CaMeL (|Δp|-based filtering) 78.33 [73.33, 82.62] CaMeL 79.33 [74.39, 83.53] LogiQA2.0 CaMeL (|Δp|-based filtering) 66.46 [63.86, 68.97] CaMeL 67.15 [64.56, 69.64] CaMeL achieves higher accuracy on both datasets. These results provide additional evidence that RCS offers benefits beyond |Δp|-based filtering.
Overall, the additional experiments provide further evidence of CaMeL’s effectiveness, demonstrating its performance gains and the contributions of its key components. We have also addressed the reviewers’ concerns through additional evaluations and analyses, and incorporated the relevant clarifications and selected results into the revised manuscript.
Thank you for your response. The explanations cleared most of my questions, especially about the sources of performance gains and the robustness of the method. Therefore, I have raised my score to 3.
Thank You for Your Consideration
Dear Reviewer,
We appreciate the time and effort you have devoted to reviewing our work. We are glad that our responses have clarified most of the concerns you raised. Should any questions or concerns remain, please feel free to let us know. We would be happy to address them.
Best regards,
The Authors
Response to Reviewer 3oRC (Part 1/2)
Summary
Thank you for your review and suggestions. To address the concerns, we conducted additional evaluations and analyses covering evaluation reliability, control specificity, and the contribution of RCS. The main results are presented below.
Evaluation Protocol and Supplementary Multi-Seed Results (W1, W4; Q1, Q4)
The test results for ReClor are unavailable because its official test server is closed. Table 3 in the original submission reports accuracy on the full development sets, including the checkpoint-selection subsets. We have kept Table 3 unchanged and conducted supplementary evaluations to examine whether the observed gains persist on evaluation data not used for checkpoint selection.
For ReClor, under the revised evaluation protocol, we use the same 200 development examples for confidence-threshold calibration and checkpoint selection, reserving the remaining 300 examples as an evaluation set. Any examples overlapping with this evaluation set are removed from the training data. For LogiQA2.0, we retain the original 627/942 calibration/selection split and additionally evaluate on the test set. Thus, the results in Table A provide additional evidence beyond the original development-set evaluation.
Due to resource and time constraints, we conducted experiments using Qwen3-4B-Instruct-2507 with three random seeds (42, 43, and 44) on ReClor and LogiQA2.0. The in-domain and OOD results are reported in Tables A and B, respectively. We also report mean accuracy with 95% confidence intervals, calculated across the three seed-level results using the Student’s t-distribution.
Table A. Accuracy on ReClor and LogiQA2.0 (Three Random Seeds: 42, 43, and 44; 95% CI)
Table B. OOD Generalization Performance (Three Random Seeds: 42, 43, and 44; 95% CI)
We find that CaMeL consistently outperforms other methods on both datasets, indicating that the observed improvements are not merely an artifact of using checkpoint-selection data for final evaluation.
To further examine the consistency of CaMeL’s gains, Table C reports per-seed and mean accuracy differences between CaMeL and GRPO-λ, as well as between CaMeL and CaMeL w/o RCS, on ReClor and LogiQA2.0.
Table C. Paired Seed-Level Accuracy Differences (Three Random Seeds: 42, 43, and 44)
Overall, these results provide additional evidence supporting CaMeL’s performance gains and the contribution of RCS. We will incorporate these results into the revised manuscript.