acceptodds
Withdrawn from ICLR 2027

When Rewards Break Traceability: Training Traceable Evidence Paths with LedgerRL

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In high-stakes settings such as medicine and finance, a reader must be able to check the evidence behind an answer: it should be correct, reach the evidence it needs, and expose small units that can be checked. Retrieval-augmented reasoners are usually trained with outcome or citation rewards and judged by accuracy or citation precision; we find that such training admits four shortcuts that break these requirements: outcome rewards leave answers without a cited path, entailment rewards encourage claims that copy their evidence, citation rewards attach many sources to broad assertions, and formats that answer in one context search little. We answer each with a matching design. A typed claim–evidence ledger lets named claims cite evidence and the answer cite claims; staged execution begins with an explicit investigation; and LedgerRL trains one shared policy with claim-level rewards that discount copied claims and count only answer-used claims toward quantity credit. On three multi-hop corpora after medical-only training, the staged ledger elicits the search that a flat citation prompt skips, and even untrained it makes more answers traceably correct than a trained citation-RL policy. Unguarded claim credit collapses traceability, whereas the guarded rewards reach 19.7% complete-path correctness, above every baseline under the same prompts, and leave fewer answers uncited than an outcome-only ledger. No arm dominates: against citation RL and a citation cap that searches about as often, LedgerRL leads at claim-level TC@2 and covers more gold evidence than the cap, but not at final-answer check units, in gold precision, or in gold-citation F1; it leaves more answers uncited and trails the cap by 7.9 points in accuracy at 1.9 times its tokens.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, traceable reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Mv4EOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduced LedgerRL, a strategy that trains one shared policy with claim-level rewards that discount copied claims and count only answer-used claims toward quantity credit. The experiments primarily utilize Qwen3-8B, employing GRPO training on MedQA questions that lack evidence labels, followed by zero-shot transfer evaluation on HotpotQA, 2WikiMultihopQA, and MuSiQue. The authors define a metric, TC@k, which requires that a given response simultaneously satisfies three conditions: the answer must be correct, the citation path must cover all gold-standard evidence, and each verification unit must cite between 1 and k sources. In this process, the study identifies and analyzes "reward shortcuts" that emerge within specific retrieval-and-citation training configurations. To address this, the authors propose the use of addressable intermediate "claim" records and protective rewards, which improve the extent to which complete evidence is incorporated into the answer path. However, it has not yet been demonstrated that these improvements consistently translate into final answers that are more accurate, easier to verify, or more cost-effective.

Critical strengths
  1. Distinguishing between "incorrect answer," "no retrieval result," and "retrieval occurred but failed to lead to the correct answer path" provides diagnostic value.

  2. Protective rewards are indeed effective within the same protocol, particularly the "anti-copy" component.

  3. There is a degree of decoupling between the training reward and the evaluation against external evidence.

  4. The authors provide a thorough disclosure of negative results and the limitations of their metrics.

Critical weaknesses

1.A simple yet powerful search rule has outperformed the full-fledged method, leaving the necessity of LedgerRL in question.

A "flat" model—devoid of reinforcement learning (RL) and augmented only by four specific search rules—surpasses the original LedgerRL across several core metrics. When both approaches are constrained to the same four search rules, "Cited-RL" achieves a TC@∞ score 5.9 percentage points higher than LedgerRL (95% CI [3.6, 8.4]). Given that such results can be attained through simple search constraints alone, what unique, irreplaceable benefits do the three-stage ledger structure and its associated RL training actually provide?


2.The advantage of primary metrics often hinges on format-specific verification units, yet training based on these metrics fails to yield consistent improvements in the verifiability of the final answers.

The C@2 metric employs different units depending on the format: for "Ledger," it uses intermediate claims cited by the answer, while for "Flat," it uses the final rationale items. A more robust verification approach should utilize semantically defined units that are consistent across all formats, while simultaneously assessing the extent to which the answer is supported by evidence.


3.While "anti-copy" is an effective regularization term in current systems, there is a lack of independent evidence regarding its utility—specifically, whether it represents a generally applicable principle for ensuring traceability.

A small-scale, validated paired analysis could be useful here: by comparing verbatim extractions (which are sometimes necessary), paraphrases that add no new information, and intermediate conclusions that genuinely synthesize multiple pieces of evidence, one could determine whether the reward mechanism makes appropriate distinctions.


4.Major concern: The existing datasets are insufficient to support the core narrative of the paper.

While the paper aims to address the issue of whether evidence can be reliably traced and verified in complex, high-stakes scenarios, the experiments primarily test whether models can locate and cite a small number of pre-annotated pieces of evidence within a short-answer question-answering (QA) framework. These two concepts are related, but they are not the same problem.

4.1 The retrieval environment is highly simplified. As explicitly stated in Appendix A, the testing does not involve open-ended searches across real-world repositories—such as medical literature, the internet, or collections of financial reports—but rather retrieval from a static BM25 index constructed from the contexts provided by the benchmark itself.

4.2 The integrity of the evidence path is reduced to a matter of "source coverage" from a predefined set of origins. In the context of multi-hop QA tasks—which are relatively short and rely on a limited number of "gold-standard" sources—this approach may measure source coverage, but it fails to adequately assess the semantic correctness of complex chains of evidence.

4.3 FinQA, which most closely resembles real-world financial document analysis, is excluded from the primary evaluation metrics. The authors explain in Appendix J that, since no method achieves an Exact Match (EM) score exceeding 3.5% on FinQA, it was omitted from the main "traceable" (TC) evaluation. Consequently, because current systems struggle to perform complex numerical reasoning based on financial documents and tables, the paper fails to demonstrate how the proposed method achieves traceable reasoning in such scenarios.

Questions for authors

Same as weaknesses

Reviewer_YJB3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how training signals for retrieval-augmented reasoners can undermine the traceability of their answers. It identifies four shortcuts: outcome rewards produce answers without a cited path (S1); entailment-based claim rewards encourage claims that copy their evidence (S2); citation rewards encourage broad assertions citing many sources (S3); and single-context formats reduce search (S4). To address these, the authors propose a typed claim-evidence ledger executed in three separately prompted stages (investigation, assessment, synthesis) of one shared policy. They also propose LedgerRL, a GRPO objective with claim-level rewards that discount lexical copying and give quantity credit only to claims used by the final answer. They introduce traceable correctness (TC@k): the fraction of answers that are correct, reach all gold supporting sources through their cited path, and require at most kk sources per check unit. Qwen3-8B is trained on MedQA without evidence labels and evaluated zero-shot on HotpotQA, 2WikiMultihopQA, and MuSiQue. LedgerRL leads the baselines on claim-level TC@2 and complete-path correctness. It ties or trails them on answer-level TC, Joint EM, gold precision, citation F1, uncited-answer rate, accuracy, and inference cost.

Critical strengths
  1. The paper addresses a relevant problem: evaluating jointly whether an answer is correct, whether its evidence path is complete, and whether each part of that path can be checked against few sources. Accuracy or citation precision alone does not capture this.

  2. The diagnosis of reward shortcuts is useful. The copy-for-entailment effect is replicated across four seeds and confirmed with a held-out judge, and the broad-citation pattern is documented across several flat citation rewards.

  3. The empirical study is extensive and transparent. It includes unguarded, outcome-only, single-context, and free-prose controls, single-guard ablations, a forced-search control, and verdict interventions. Negative and mixed results are reported openly, and the limitations are stated explicitly.

  4. The one-claim support-withdrawal intervention (Sec. 6) is a well-designed test of whether generated answers depend on the recorded claim layer.

Critical weaknesses
  1. The headline metric depends on the format. TC@2 uses ledger claims as check units for the proposed method but final rationale items for the flat baselines, so the comparison favours the ledger. Under format-neutral variants reported in the paper, the advantage disappears. At the common answer-level unit, LedgerRL ties CITED-RL (+0.3, [-1.5, 2.0]) and the citation cap (-0.5, [-2.7, 1.4]) (Table 7). Under precision conditioning or a bound on total sources, the gap to the citation cap is about zero (Table 12). Joint EM shows no difference.

  2. Training contributes little beyond the untrained staged prompt. Compared with the untrained staged ledger, LedgerRL's TC@2 gain is not significant (+1.5, [-0.6, 3.6]). Answer-level TC@2 (-1.9, [-3.9, -0.1]) and gold precision (-4.1) are significantly worse (Tables 7-8). The guards mainly prevent a collapse caused by the unguarded claim reward itself. LedgerRL's verbatim overlap (0.342) also remains higher than the untrained model's (0.229, Table 2), so "halving copying" holds only relative to the unguarded arm.

  3. The method does not address S3 at the level a reader checks. About 30% of LedgerRL's final rationale items reach more than two sources (Table 13; 18.9-43.4% across seeds). That exceeds CITED-RL's 24%, the broad-citation pattern the paper criticises. Fine-grained check units exist only at the intermediate claim layer.

  4. The staged protocol's advantage comes from eliciting search, which a simple rule matches or exceeds. With an external four-search requirement (Table 19), untrained flat citation reaches 19.3% TC@2 vs 13.8% for the untrained staged ledger. CITED-RL also exceeds LedgerRL by 3.5 points at TC@2 and 5.9 at complete-path TC. The untrained comparison is further confounded by prompt wording: the investigator is instructed to "search, then post claims," whereas the flat prompt asks the model to answer "when confident."

  5. The accuracy and cost penalty is substantial. LedgerRL averages 0.635 on full MedQA, vs 0.741 (Search-RL), 0.681 (CITED-RL), and 0.717 (free-prose stages). Out of domain it trails the citation cap by 7.9 EM points while using 1.9x its tokens (3.9x CITED-RL's). Absolute TC levels remain low (about 15%).

  6. Statistical and experimental support is limited. Arms have only 2-3 seeds, and the guard ablations are single-seed. The bootstrap intervals are conditional on checkpoints and ignore seed variance, which is large: the share of broad items ranges from 18.9 to 43.4%, and the sign of the semantic-support comparison depends on the seed (Table 25). One citation-cap baseline gives no answer on 590 of 1,273 MedQA questions, which weakens it as a comparison point. Compute is not matched across protocols, since staged training regenerates all upstream stages for each sample. TC is evaluated only zero-shot after medical-only training, so the setting the method targets is never trained on directly.

  7. Conceptual novelty is limited. S2 is an instance of the known lexical-overlap reliance of NLI models and of reward hacking more generally, and TC@k is closely related to HotpotQA Joint EM. The reward design rests on hand-tuned lexical heuristics (overlap thresholds 0.40/0.80, Jaccard 0.60) and several manually set weights. The closest prior work on traceability rewards (Ren et al., 2025) and Self-RAG are discussed but not compared empirically.

  8. Clarity and presentation need work. The paper is very dense. The abstract and introduction are hard to follow because of heavily qualified claims across many arms and metrics, and key evidence is deferred to a long appendix. Notation needs tightening:

    • Eq. (1) mixes an entailment relation (⊨\models) with a score threshold (≥τ\geq \tau); a score function should be defined and then thresholded.
    • mm denotes assertions in Eq. (1) but claims in Eqs. (3)-(5).
    • The assessment pass is called the "auditor" in the appendix (raudr_{aud}); rinvr_{inv}, raudr_{aud}, rsynr_{syn} appear only in Appendix C.
    • The effective copy threshold implied by giκ(vi)≥1/2g_i \kappa(v_i) \geq 1/2 (i.e., vi≤0.6v_i \leq 0.6) is not stated.
    • Sec. 2 should likely relate Joint EM to TC@∞\infty rather than TC@1 restricted to gold precision one.

    The same quantities are reported with different values depending on 3- vs 4-corpus averaging (e.g., searches per question, EM gap to the cap). MedQA results come from two subsets with an unexplained gap (untrained accuracy 0.695 on n=200 vs 0.589 on the full test set), and how the subset was selected is not specified.

Overall assessment. The paper's diagnosis of attribution shortcuts and its transparent reporting are valuable. However, as a method paper, its central claim is not supported by its own controls: the gains of LedgerRL do not survive format-neutral evaluation, are not significant over the untrained prompt, and are matched or exceeded by a simple search rule, all at a notable accuracy and compute cost. My recommendation rests mainly on Weaknesses 1-4. Reframing the work as an analysis of reward shortcuts and the limits of path-based attribution metrics may be a more convincing contribution in a future version.

Questions for authors

I have no questions beyond the points raised under Weaknesses. My recommendation rests mainly on Weaknesses 1-4, which follow from the paper's own reported results (Tables 7, 12, 13, 19); I do not expect clarifications alone to resolve them.

One clarification: how was the MedQA n=200 subset selected, given the gap between subset and full-test-set accuracy (0.695 vs 0.589 for the untrained model)?

Reviewer_bFLdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies four reward-induced shortcuts in retrieval-augmented reasoning and proposes LedgerRL, combining a typed claim–evidence ledger, staged execution, and guarded claim-level RL rewards. It introduces TC@k to jointly evaluate answer correctness, evidence-path completeness, and citation granularity.

The failure-mode analysis is valuable, but the experiments do not convincingly establish that LedgerRL improves end-to-end traceability over simpler alternatives. Several closely related methods are also missing from the literature discussion.

Critical strengths
  • Useful reward-hacking analysis. The copying failure under unguarded entailment rewards is supported across multiple seeds (Table 20), and broad citation units are observed under several citation-reward variants (Table 14).

  • Careful empirical analysis. The authors report unfavorable results, multiple TC variants, reward ablations, held-out entailment judges, and interventions on intermediate claim verdicts.

  • Potentially useful representation. An explicit, addressable answer → claim → evidence structure provides a foundation for evidence auditing and targeted correction.

Critical weaknesses

The paper offers a rigorous investigation of reward shortcuts. Its improvements in evidence-path completeness are meaningful. However, the central empirical advantage is sensitive to the evaluation unit and retrieval protocol, while accuracy, efficiency, and final-answer grounding remain unresolved. The omission of closely related faithfulness-RL methods also weakens the novelty case.

  1. A simple search-control baseline challenges the main improvement. Appendix C (Table 19) shows that requiring four searches raises untrained flat citation TC@2 from 4.8% to 19.3%, exceeding trained LedgerRL's native 15.3%. Under the same four-search requirement, Cited-RL also exceeds LedgerRL (16.2% vs. 12.8%). This suggests that much of the benefit comes from eliciting retrieval rather than the ledger or its reward design. The four-search baseline deserves greater prominence. I recommend comparing methods under fixed retrieval budgets, ideally with identical retrieved evidence and matched inference costs, and investigating whether training Cited-RL with a minimum-search constraint yields similar benefits.
  2. The primary metric is sensitive to representation and citation precision. LedgerRL leads the citation cap on claim-level TC@2 (15.3% vs. 11.1%), but the advantage disappears using shared final-answer check units (10.6% vs. 11.1%). Requiring gold citation precision ≥ 0.75 also reduces the difference to +0.4 points, with a confidence interval including zero (Table 12). This raises a concern that TC@2 partly rewards the ledger's claim segmentation rather than improved verification of final answers. I recommend making format-neutral answer-level metrics more central and reporting a joint criterion incorporating semantic support and citation precision.
  3. Training improves evidence coverage but not consistently final-answer quality. Relative to the untrained staged ledger, LedgerRL improves TC@2 by only +1.5 points [−0.6, 3.6], while answer-level TC@2 decreases by 1.9 points. The +5.1-point improvement in TC@∞ is meaningful and should be acknowledged, but it primarily establishes better evidence-path completeness. Meanwhile, approximately 30% of LedgerRL's final rationale items reach more than two sources, compared with 24% for Cited-RL (Table 13), and 18.2% of answers remain uncited versus 6.2% for Cited-RL. LedgerRL also trails the citation cap in out-of-domain accuracy (37.1% vs. 45.0%) while using approximately 1.9× its inference tokens. These trade-offs weaken the claim that guarded rewards resolve the identified shortcuts at the final-answer level.
  4. Grounding and faithfulness are not sufficiently established. The anti-copy reward penalizes the longest shared contiguous token run. This is a lexical heuristic: paraphrasing could evade the penalty without necessarily producing a more informative or faithful claim. CITEP_abs is also part of LedgerRL's training objective, making improvements on that metric insufficient as independent validation. I suggest testing semantic copying, paraphrase robustness, and claim atomicity using independent judges or human annotations. Counterfactual evidence replacement would also provide a stronger test of whether answers genuinely depend on retrieved evidence rather than parametric knowledge.
  5. Experimental reliability and practical validation remain limited. The citation-cap baseline exhibits a major failure in seed 1234, producing no valid final answer for 590 of 1,273 MedQA questions (Appendix H). The authors should clarify whether this reflects an implementation issue, formatting instability, or reward-induced behavior, and verify that the comparison remains robust after addressing it. Additionally, evaluation uses benchmark-derived retrieval pools, training-seed coverage is limited, and the staged protocol is not compute-matched. Despite motivating the work through reduced verification burden, no human verification study is provided.

Missing related work

Several relevant studies are not cited:

  • Context-DPO (https://aclanthology.org/2025.findings-acl.536/) studies context-faithfulness under knowledge conflicts. Its evaluation setting motivates testing whether the ledger actually controls answers when retrieved evidence contradicts parametric knowledge.
  • TruthRL (https://arxiv.org/pdf/2509.25760) optimizes truthful answering and abstention. It raises an important complementary question: how should traceability be evaluated when the available evidence is insufficient to justify an answer?
  • EvidenceRL (https://arxiv.org/pdf/2603.19532) uses GRPO with evidence-entailment rewards in high-stakes applications. Testing whether its grounding objective exhibits the copying shortcut identified here would strengthen the paper's generality.

The authors should distinguish LedgerRL's contributions—particularly its addressable ledger, anti-copy regularization, and answer-used claim gating—from existing process-level faithfulness and grounding rewards.

Questions for authors

See the weakness section above + additional questions below:

  • What improvement specifically comes from the typed ledger and guarded rewards after controlling for staged prompting and additional search?
  • Does the anti-copy guard remain effective against semantic paraphrasing, and can counterfactual evidence interventions demonstrate genuine evidence dependence?
  • Can the anomalous citation-cap run and the role of additional assessment-stage training be investigated? Could a small human study validate whether TC@k predicts actual verification effort?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.