- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper addresses credit assignment for RLVR-trained LLM agents on longitudinal EHR tasks with bounded memory and an action budget. It argues that trajectory-level GRPO cannot tell which action earned a reward. The proposed VAPA has three parts: 1) 14 deterministic verifiers that score retrieval, memory, and stopping actions. 2) Step groups built by forked replay of logged prefixes, with observation collisions and a same-turn-index pool as fallbacks. 3) A leave-one-out local advantage added to an episode advantage. Trained on MIMIC-IV with Qwen3.5-9B (plus gpt-oss-20b) and transferred to MedAgentBench and EHRSHOT, VAPA has the highest mean on all six endpoints over about 14 baselines. Validation includes a 2×2 factorial, a token-matched no-fork control, a memory-capacity sweep, a depth analysis, and verifier audits.
Critical strengths
- Rigorous experiments. The paper uses five seeds, a factorial design with interaction tests, Holm-corrected paired intervals, matched token budgets, held-out transfer, and a second backbone. This makes the core finding credible: process rewards and step credit are super-additive (interaction of +3.53 pp on grounded EM and +5.47 pp on MedAgentBench).
- Informative ablations. The memory sweep shows the margin shrinking as evictions vanish. Together with the depth trend and the matched-budget replay control, it links the gain to the mechanism rather than to tuning.
Critical weaknesses
W1. Writing obscures the contribution.
- The introduction is hard to follow, especially the last paragraph.
- Figs. 1–2 do not clearly show how VAPA’s credit differs from GRPO’s on one trajectory.
- Section 3.2 crams three grouping tiers and several undefined terms into a few paragraphs. Key details (G0, K, fork selection, loss, masking) sit in the appendix.
- Table 13 shows exact forks cover only ~3% of turns and the turn-index pool ~70%. Most “local” credit is therefore close to episode credit, not the same-state comparison the figures depict.
- Novelty is never stated. The episode + step advantage over state-matched groups follows GiGPO. Forked continuations follow VinePPO, Tree-GRPO, and LoGo-GRPO. Verifiable process rewards follow VPR, and the reward formulation is standard.
W2. Scope and fairness.
- The method is domain-agnostic, yet only EHR is evaluated. Standard agent benchmarks (ALFWorld, WebShop, search QA) would test generality.
- Several verifiers rely on reference-program knowledge, a privileged signal that outcome-only baselines lack. The control “GiGPO plus the same verifiers” is missing.
- Each component alone trails existing baselines: A2 is below GiGPO on five of six endpoints, and A3 is below Tree-GRPO + process on all six.
- VAPA is tuned on an 18-config grid versus 5 points per baseline.
- Against VinePPO and Tree-GRPO + process, the calculation and MedAgentBench differences are not significant.
- The token budget excludes replayed-prefix, tool, and verifier compute. VAPA’s ~20% GPU-hour overhead over GRPO is not discussed in the main text, and per-baseline costs are not reported.
Questions for authors
Clarifications:
- What is new relative to GiGPO, VinePPO, Tree-GRPO, LoGo-GRPO, and VPR?
- How does GiGPO perform with the same verifier rewards?
- What share of local credit comes from each tier? Does the gain hold at full budget without the turn-index pool?
- Which verifiers use reference-program information, and how does VAPA perform without them?
- Do the rankings hold when methods are matched on GPU-hours and given equal tuning budgets?
Suggestions:
- Add one non-EHR benchmark.
- Restructure Sec. 3.2 and Fig. 1, and move the key settings and Table 13 into the main text.
4 comments
Dear Reviewer 3dHd,
Thanks again for the detailed review. We took your comments on clarity seriously: Section 3.2 has been restructured, and the introduction now states the contribution in one sentence. Appendix C.2 also has the comparisons you asked for (GiGPO with the same verifiers, the two VAPA ablations, and per-run GPU-hours).
If anything is still unclear, we're happy to discuss it before the deadline.
Best regards,
Authors of Paper 7868
Response to Rebuttal
Thank you for the thoughtful rebuttal and substantial revisions. The improved presentation and additional experiments, particularly GiGPO with the same verifiers, the ablations on grouping and privileged supervision, and the compute analysis, have addressed most of my concerns. I am now more confident in the experimental findings and the effectiveness of the proposed framework. My remaining reservations concern the incremental methodological novelty relative to existing credit assignment approaches and the limited generality beyond EHR tasks, especially given the reliance on reference-derived verifiers. Overall, I appreciate the authors' efforts and have raised my rating to 3 (Weak Accept).
Thank you for reconsidering and for raising your rating. We agree that generality beyond EHR and the reliance on reference-derived verifiers are the main open points, and the revised Limitations section states both. Your comments made the paper substantially clearer, and we appreciate the time you put into the review. Thanks again!
We thank Reviewer 3dHd for the close reading. We agree with most points and revise the paper accordingly. We also add GiGPO with the same verifiers and two VAPA ablations, clarify tuning budgets, and report compute costs.
W1. Writing and contribution
W2. Scope and fairness
Five seeds at the full token budget. GiGPO + process and both VAPA ablations are new, the other accuracies are from Table 1, and all GPU-hours are in Table 19.
VAPA keeps the highest means, and Tree-GRPO + process, the strongest baseline, uses more GPU-hours. Removing the turn-index pool or the reference-derived predicates lowers VAPA's means, the latter below the strongest outcome-only baselines (Table 1), and the paper now reports both.
Questions. Q1 is covered in W1.5, Q2 and Q4 in W2.2 and the table, Q3 in W1.4 and the no-pool row, and Q5 in W2.4 and W2.6. Appendix C.2 lists the removed predicates.
Suggestions. We thank the reviewer for both suggestions. S1 is acknowledged in W2.1, and the revision makes the S2 changes.