- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work investigates credit assignment difficulties encountered when training reinforcement learning agents for temporal knowledge graph question answering (TKGQA) tasks. The authors propose CREDO (Credit Redistribution with Evidence-Driven Opportunities), which uses intermediate evidence to recover local supervision signals even when all rollouts for a question fail. CREDO assigns positive stage-level credit through within-question rollout comparisons and uses evidence opportunity proxies to shift outcome policy weight toward reasoning and answering stages. Experiments on MultiTQ and TimelineKGQA datasets show significant improvements over baseline methods, particularly for multi-hop temporal reasoning tasks.
Critical strengths
Critical Strengths
1.The paper addresses an important credit assignment problem in reinforcement learning for language agents.
2.The idea of using intermediate evidence as a proxy for partial credit is conceptually appealing.
Critical weaknesses
Critical Weaknesses
1.Evaluation limitations: The paper only evaluates on two small temporal KG QA datasets, and the reported improvements are marginal (less than 5% absolute) compared to strong GRPO baselines with proper hyperparameter tuning. The improvements disappear when using larger backbone models, suggesting the approach has limited practical value.
2.Theoretical inconsistencies: The credit redistribution mechanism violates the policy gradient theorem by arbitrarily shifting weights between stages without proper theoretical justification, which can lead to unstable training dynamics not reported in the paper.
3.Lack of ablation analysis: The paper does not include sufficient ablation studies to isolate the contribution of the credit assignment mechanism from other confounding factors in the training pipeline.
4.Limited generalizability: The approach is highly specialized to TKGQA tasks with explicit intermediate evidence markers, and cannot be easily extended to general agent tasks where intermediate evidence is not clearly defined.
Questions for authors
None
3 comments
Response to Reviewer
We continue our response with the component ablations and applicability beyond TKGQA.
Weakness3: Component ablations and attribution of the observed gains
The main 8B experiments include component comparisons examining local credit, redistribution target positions, and evidence-conditioned eligibility:
| Variant | Configuration | Overall (%) | CREDO gain (pp) |
|---|---|---|---|
| GRPO | Outcome-only supervision | 77.0 | +5.4 |
| CWO | Local credit without outcome redistribution | 80.2 | +2.2 |
| Random | Local credit with randomized redistribution targets | 79.0 | +3.4 |
| No-gate | Local credit and redistribution without evidence-overlap eligibility | 78.4 | +4.0 |
| CREDO | Full evidence-conditioned configuration | 82.4 | — |
These comparisons share base training settings, the checkpoint-selection criterion, and final evaluation settings. Random preserves redistribution eligibility, transferred weight, and target-token count while changing the target positions. No-gate removes the evidence-overlap requirement for redistribution while retaining structural and length eligibility; its local-credit mechanism remains unchanged.
The full configuration achieves the highest observed accuracy across these comparisons. Component results and the corresponding paired analyses are reported in Appendix A.3, including Table 6.
The original CREDO training trace separately demonstrates the availability of intermediate supervision within the same all-wrong question groups:
| Learning signal in all-wrong groups | Groups with a nonzero signal | Proportion |
|---|---|---|
| Outcome-derived GRPO advantage | 0/1,473 | 0.00% |
| CREDO stage-local credit | 564/1,473 | 38.29% |
This directly demonstrates that the intended intermediate learning signal remains available when outcome advantages are zero. The original trace and counts appear in Figure 3 and Appendix A.3 ("All-wrong groups").
Weakness4: Applicability beyond TKGQA
CREDO targets multi-stage evidence-seeking agents, with TKGQA as the experimental instantiation. Its local-credit and redistribution rules operate on stage quality, upstream evidence opportunity, and stage-token masks; these rules do not depend on a particular knowledge-graph schema or temporal operator.
A concrete extension is extractive document QA, where the required inputs can be instantiated as follows:
| CREDO interface | TKGQA instantiation | Extractive document QA instantiation |
|---|---|---|
| Retrieved evidence | Retrieved KG facts | Retrieved passages |
| Retained evidence | Selected facts | Selected passages |
| Evidence-opportunity proxy | Reference-answer coverage in facts | Reference-answer coverage in passages |
| Answering-stage quality | Generated answer compared with references | Generated answer compared with references |
| Stage-token masks | Retrieval, selection, and answering spans | Corresponding spans in the document-QA trace |
The same reference-answer coverage definitions can be applied to passages, allowing the within-question comparisons and outcome-redistribution rules to be instantiated without changing their core formulation.
The required interface is observable intermediate evidence and identifiable stage boundaries. In our implementation, these boundaries come from the agent's interaction trace and do not require additional human annotations of intermediate-step correctness. Thus, applicability depends on the structure and observability of the agent workflow. CREDO is a credit-assignment framework for staged evidence-seeking agents, instantiated and evaluated here on TKGQA.
We hope these responses address the concerns raised and that the reviewer will reconsider the assessment.
Dear Reviewer p8xK,
We are writing to kindly follow up on our rebuttal submitted a few days ago. We would appreciate your feedback on whether our responses address your concerns.
Thank you very much for your time and continued effort in reviewing our work. We look forward to any further feedback you may have.
Best regards.
Response to Reviewer
We sincerely thank the reviewer for the careful reading of our manuscript and the constructive feedback. We have revised the manuscript and added supplementary experiments and analyses to address the concerns raised. Changes in the revised PDF are highlighted in blue. The weighted-objective clarification and coefficient derivation are in Section 3.4 and Appendix A.1; the main 8B results and paired comparisons are in Appendix A.3 (Tables 5 and 6). The additional 3B method comparison is reported here in this response.
Weakness1: Evaluation scope, improvement magnitude, and backbone size
Our main benchmark and component experiments use Meta-Llama-3.1-8B-Instruct. The reproduced outcome-only GRPO baseline shares CREDO's SFT initialization, training data, terminal reward, and base training and evaluation settings. We distinguish this matched comparison from the published Temp-R1 reference:
We additionally report a 3B method comparison, selecting each method's highest-Overall checkpoint among the reported checkpoints. All category-level results come from the same selected checkpoint for each method. Accuracy values are percentages.
The main gains are already obtained with the larger 8B backbone. Both reported model scales show positive Overall gains in their respective experimental settings, directly addressing the concern that the improvement disappears on the larger backbone.
Weakness2: Optimization objective, policy gradients, and training control
CREDO optimizes an explicitly defined evidence-conditioned weighted PPO surrogate:
J(θ)=Joutcomew(θ)+λlocalJlocal(θ).
The outcome term retains the original terminal rewards and trajectory-level GRPO advantages, while its token weights specify where that supervision acts. The local term supplies the additional stage-level learning signal.
Redistribution changes the surrogate being optimized; it does not assert that stage reweighting leaves the original unweighted policy gradient unchanged. Weight conservation controls the allocation budget, while the weighted objective explicitly defines the optimization update.
Here, f is the reasoning/answering fraction of trainable tokens. These bounds prevent unbounded weight concentration on a short reasoning span. The implementation also retains PPO ratio clipping and KL regularization. Sections 3.3-3.4 and Appendix A.1 define the objective and allocation controls; Figure 5(a) and Table 3 provide training-stage evaluations and coefficient-sensitivity diagnostics. Table 3 concerns the existing sensitivity study, while the additional 3B method comparison is given above.