acceptodds
Under review as a conference paper at ICLR 2027

Where Credit Lands: Verifiable Step Credit for Bounded-Memory EHR Agents

Abstract

Reinforcement learning with verifiable rewards (RLVR) has enabled large language model (LLM) agents to solve clinical tasks over longitudinal electronic health records (EHRs), where an agent takes a sequence of retrieval and memory decisions over evidence scattered across years of visits. However, the usual optimizer, group relative policy optimization (GRPO), reduces each rollout to one score and gives every action the same advantage, so it cannot tell which action helped or hurt. Crediting each action needs its own score and a comparison with its alternatives, neither of which is readily available for current EHR agents. We propose Verifier-Anchored Process Advantage (VAPA), a critic-free training method that pairs checks of individual actions with comparisons of alternative decisions, needs no learned reward model, and uses neither verifiers nor replay at test time. First, deterministic checks score retrieval, memory, and stopping actions from the patient record, completed rollout, and task specification. Second, forked replay rewinds to the state before a selected action and samples alternative continuations, with weaker fallback groups where no fork reaches. Finally, a two-level advantage assigns local credit from these comparisons while preserving final-answer credit for every original action. Across MIMIC-IV calculation and retrieval tasks and the held-out MedAgentBench and EHRSHOT suites, VAPA has the highest mean among matched agent, memory, and RL baselines, with no further training for transfer. At the same sampled-token budget, replaying alternative decisions outperforms additional independent rollouts, and VAPA's lead over trajectory-level GRPO grows with decision-chain length and with a tighter memory bound.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
credit assignment, reinforcement learning, process rewards, LLM agents, verifiable rewards, advantage estimation, electronic health records, clinical agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3dHdOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses credit assignment for RLVR-trained LLM agents on longitudinal EHR tasks with bounded memory and an action budget. It argues that trajectory-level GRPO cannot tell which action earned a reward. The proposed VAPA has three parts: 1) 14 deterministic verifiers that score retrieval, memory, and stopping actions. 2) Step groups built by forked replay of logged prefixes, with observation collisions and a same-turn-index pool as fallbacks. 3) A leave-one-out local advantage added to an episode advantage. Trained on MIMIC-IV with Qwen3.5-9B (plus gpt-oss-20b) and transferred to MedAgentBench and EHRSHOT, VAPA has the highest mean on all six endpoints over about 14 baselines. Validation includes a 2×2 factorial, a token-matched no-fork control, a memory-capacity sweep, a depth analysis, and verifier audits.

Critical strengths
  1. Rigorous experiments. The paper uses five seeds, a factorial design with interaction tests, Holm-corrected paired intervals, matched token budgets, held-out transfer, and a second backbone. This makes the core finding credible: process rewards and step credit are super-additive (interaction of +3.53 pp on grounded EM and +5.47 pp on MedAgentBench).
  2. Informative ablations. The memory sweep shows the margin shrinking as evictions vanish. Together with the depth trend and the matched-budget replay control, it links the gain to the mechanism rather than to tuning.
Critical weaknesses

W1. Writing obscures the contribution.

  1. The introduction is hard to follow, especially the last paragraph.
  2. Figs. 1–2 do not clearly show how VAPA’s credit differs from GRPO’s on one trajectory.
  3. Section 3.2 crams three grouping tiers and several undefined terms into a few paragraphs. Key details (G0, K, fork selection, loss, masking) sit in the appendix.
  4. Table 13 shows exact forks cover only ~3% of turns and the turn-index pool ~70%. Most “local” credit is therefore close to episode credit, not the same-state comparison the figures depict.
  5. Novelty is never stated. The episode + step advantage over state-matched groups follows GiGPO. Forked continuations follow VinePPO, Tree-GRPO, and LoGo-GRPO. Verifiable process rewards follow VPR, and the reward formulation is standard.

W2. Scope and fairness.

  1. The method is domain-agnostic, yet only EHR is evaluated. Standard agent benchmarks (ALFWorld, WebShop, search QA) would test generality.
  2. Several verifiers rely on reference-program knowledge, a privileged signal that outcome-only baselines lack. The control “GiGPO plus the same verifiers” is missing.
  3. Each component alone trails existing baselines: A2 is below GiGPO on five of six endpoints, and A3 is below Tree-GRPO + process on all six.
  4. VAPA is tuned on an 18-config grid versus 5 points per baseline.
  5. Against VinePPO and Tree-GRPO + process, the calculation and MedAgentBench differences are not significant.
  6. The token budget excludes replayed-prefix, tool, and verifier compute. VAPA’s ~20% GPU-hour overhead over GRPO is not discussed in the main text, and per-baseline costs are not reported.
Questions for authors

Clarifications:

  1. What is new relative to GiGPO, VinePPO, Tree-GRPO, LoGo-GRPO, and VPR?
  2. How does GiGPO perform with the same verifier rewards?
  3. What share of local credit comes from each tier? Does the gain hold at full budget without the turn-index pool?
  4. Which verifiers use reference-program information, and how does VAPA perform without them?
  5. Do the rankings hold when methods are matched on GPU-hours and given equal tuning budgets?

Suggestions:

  1. Add one non-EHR benchmark.
  2. Restructure Sec. 3.2 and Fig. 1, and move the key settings and Table 13 into the main text.
4 comments

We thank Reviewer 3dHd for the close reading. We agree with most points and revise the paper accordingly. We also add GiGPO with the same verifiers and two VAPA ablations, clarify tuning budgets, and report compute costs.


W1. Writing and contribution

  • W1.1. The revision shortens the limitations paragraph, states the contribution in one sentence before the components, and moves the results into a separate paragraph.
  • W1.2. Figure 1B shows GRPO's advantage, the same value at every step, and the revised caption names GRPO so that it can be compared with VAPA's local advantage at t4 in Figure 1C.
  • W1.3. Section 3.2 now gives one paragraph to each grouping tier and defines every term at first use. G0, K, fork selection, the loss, and masking move to the main text.
  • W1.4. Exact forks cover about 3% of base turns by design. On turn-index pools the outcome part of the local advantage is an episode-style contrast within the participating rollouts, so those turns differ from episode credit mainly through process and cost scores (Appendix B.4). Section 3.2 now states these shares, and the no-turn-pool ablation below shows what removing the pool costs.
  • W1.5. Each component has precedent in GiGPO, VinePPO, Tree-GRPO, LoGo-GRPO, and VPR. Our contribution is scoring forked alternatives with deterministic verifiers inside same-state groups and combining this local credit with episode credit. In our factorial, verifier rewards and step-level credit are super-additive (Section 4.3).

W2. Scope and fairness

  • W2.1. VAPA is designed for EHR agents. Forked replay and the two-level advantage could be adapted to tasks whose tools fit the action interface and whose relevant pre-action state can be restored, given task-specific checks. Testing this outside EHR remains future work.
  • W2.2. The verifiers read reference-derived fields such as windows and evidence requirements, which are privileged relative to outcome-only baselines, but never the gold answer, and this metadata is not added to the policy observation (Appendix A.2). Tree-GRPO + process and adapted VPR train with the same verifiers (Appendix B.3), and we add GiGPO with them below.
  • W2.3. A2 and A3 are single-factor ablations run at the coefficients selected for full VAPA, while GiGPO and Tree-GRPO + process are tuned methods built around one of these factors. The factorial tests whether the two factors work together, and their interaction is positive on every endpoint (Table 2).
  • W2.4. VAPA searches three coefficients jointly with two or three values each, while each baseline searches one coefficient over five values (Table 10), so the tuning budgets are not equal. The values were selected for full VAPA and then frozen for every factorial arm and for gpt-oss-20b, where VAPA still leads GRPO on every endpoint (Table 1).
  • W2.5. The significant gains over these comparators are on grounded retrieval and both EHRSHOT endpoints (Appendix C.1), and the revision presents calculation and MedAgentBench task success as higher means.
  • W2.6. The table reports GPU-hours per run, including replayed prefixes, tools, and verifiers. VAPA costs about 20% more than GRPO (Table 14), and Section 4.3 now reports this next to the token-matched results.
MethodGPU-hours per runGrounded EM (%)MedAgentBench-Query success (%)
GRPO8.6970.2132.53
GiGPO9.2273.6636.13
GiGPO + process9.7076.2138.27
Tree-GRPO + process10.8176.0640.53
VAPA10.3978.0342.67
VAPA without turn-index pool10.2376.7140.67
VAPA without reference-derived predicates9.9274.9938.93

Five seeds at the full token budget. GiGPO + process and both VAPA ablations are new, the other accuracies are from Table 1, and all GPU-hours are in Table 19.

VAPA keeps the highest means, and Tree-GRPO + process, the strongest baseline, uses more GPU-hours. Removing the turn-index pool or the reference-derived predicates lowers VAPA's means, the latter below the strongest outcome-only baselines (Table 1), and the paper now reports both.


Questions. Q1 is covered in W1.5, Q2 and Q4 in W2.2 and the table, Q3 in W1.4 and the no-pool row, and Q5 in W2.4 and W2.6. Appendix C.2 lists the removed predicates.

Suggestions. We thank the reviewer for both suggestions. S1 is acknowledged in W2.1, and the revision makes the S2 changes.

Dear Reviewer 3dHd,

Thanks again for the detailed review. We took your comments on clarity seriously: Section 3.2 has been restructured, and the introduction now states the contribution in one sentence. Appendix C.2 also has the comparisons you asked for (GiGPO with the same verifiers, the two VAPA ablations, and per-run GPU-hours).

If anything is still unclear, we're happy to discuss it before the deadline.

Best regards,

Authors of Paper 7868

Reviewer_3dHdOpenReview ↗

Response to Rebuttal

Thank you for the thoughtful rebuttal and substantial revisions. The improved presentation and additional experiments, particularly GiGPO with the same verifiers, the ablations on grouping and privileged supervision, and the compute analysis, have addressed most of my concerns. I am now more confident in the experimental findings and the effectiveness of the proposed framework. My remaining reservations concern the incremental methodological novelty relative to existing credit assignment approaches and the limited generality beyond EHR tasks, especially given the reliance on reference-derived verifiers. Overall, I appreciate the authors' efforts and have raised my rating to 3 (Weak Accept).

Thank you for reconsidering and for raising your rating. We agree that generality beyond EHR and the reliance on reference-derived verifiers are the main open points, and the revised Limitations section states both. Your comments made the paper substantially clearer, and we appreciate the time you put into the review. Thanks again!

Reviewer_Dy5AOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses credit assignment for RLVR/GRPO-trained bounded-memory EHR agents, where every action in a rollout receives the same trajectory-level advantage. It proposes VAPA, a critic-free method that (i) scores retrieval, memory, and stopping actions with deterministic record-grounded checks, (ii) forms same-state comparisons by replaying a logged prefix and sampling alternative continuations, with canonical-collision and turn-index groups as fallbacks, and (iii) combines a leave-one-out local advantage with an episode advantage shared by all base actions. Trained on MIMIC-IV and transferred frozen to MedAgentBench and EHRSHOT, VAPA has the highest mean on all six endpoints.

Note that the methodological novelty of this paper lies primarily in combining several existing credit-assignment methods, with some additional innovations.

Critical strengths
  1. A fully specified, critic-free estimator whose reward scale is principled and outcome-dominant by construction.
  2. A matched-token finding that forked replay is a better use of the sampling budget than extra independent rollouts.
  3. A rigorous empirical protocol: pre-specified confirmatory tests, negative controls, coverage accounting, audits, and stress tests.
Critical weaknesses

W1. The method assumes a deterministic environment, and its verifiers are hand-built. Forked replay requires that replaying the prefix recovers the same observation which is guaranteed only by the study's constructed environment, not by realistic tool environments (e.g., approximate retrieval, failed tool calls, LLM-based tools). The 14 predicates are also hand-authored against the task manifest and validated by clinician annotation, a human cost absent from the compute ledger and not easy to transfer to new task families.

W2. There is no ablation of the number of fork anchors. The schedule (at most 22 anchors per group, K=3,G0=8K=3, G_0=8) is fixed a priori; branch rollouts take 28% of tokens while exact forks cover only 2.96% of base turns. Among the fork-schedule parameters, only KK and G0G_0 are ablated, so "replay beats extra rollouts" rests on a single 0%/28% comparison with no curve showing how much replay is actually needed.

Questions for authors

Please address my concerns regarding W1 and W2.

If the authors can clarify these points, I believe the paper would be significantly strengthened, and I would consider raising my score.

2 comments

We thank Reviewer Dy5A for questioning whether forked replay needs a deterministic environment and how much replay is needed. We clarify the replay requirement and the verifier cost, and add a fork-anchor sweep at matched tokens.


W1. Deterministic environment and hand-built verifiers

  • Replay. Forked replay does not inherently require a deterministic environment. It needs the pre-action state, which our implementation recovers by deterministic replay (Section 2). The agent's observation and memory are built from the logged actions and tool returns, so for tools whose relevant state can be saved or held fixed, the fork could restore that state from the log instead. In web-search QA, for example, the fork would reload the logged pages and memory, and only the sibling continuations would search again. Section 3.2 now states this scope.
  • Verifiers. RL rewards are computed automatically and need no clinician labels. Clinicians audited the verifiers only after they were frozen (1,700 action–verifier opportunities and 1,200 applied penalties, Appendix E.1), and the compute ledger in the revision now lists this effort (Appendix B.5). Record-grounding, unit, and temporal predicates can be reused for new EHR tasks with schema adaptation, while evidence-requirement and stopping rules need task-specific adaptation.

W2. Number of fork anchors

  • Coverage. For VAPA, exact forks cover 3.28% of base turns and branches take 28.36% of tokens. The 2.96% in Appendix B.4 pools VAPA with A2. Coverage is low because each base group allows at most two anchors, and the token share is high because each anchor runs three continuations to termination.
  • Amount of replay. We vary the maximum number of anchors per group at the same total sampled tokens.
Max anchorsBranch tokens (%)Exact-fork coverage (%)Grounded EM (%)MedAgentBench-Query success (%)
No fork0.000.0073.5237.60
118.351.5876.8140.27
228.363.2878.0342.67
448.896.3277.7545.47

A4 at the full token budget, with the five seeds of Table 12. The 1, 2, and 4 rows fix G0 = 8, and the no-fork row is the Table 12 control, which spends branch tokens on enlarged base groups.

Within the sweep, the highest grounded EM mean is at two anchors and the highest MedAgentBench mean is at four. We added the sweep to Appendix C.2.

Dear Reviewer Dy5A,

Thanks again for the review. Your question about determinism made us state more carefully what forked replay actually needs, and the anchor sweep you asked for is now in Appendix C.2.

If our reply resolves W1 and W2, we'd be grateful if you would consider updating your score. And if anything is still unclear, we're happy to discuss it before the deadline.

Best regards,

Authors of Paper 7868

Reviewer_BLQdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates fine-grained credit assignment in reinforcement learning-based EHR agents, addressing the limitations of trajectory-level GRPO in distinguishing the contributions of intermediate actions. The authors propose Verifier-Anchored Process Advantage (VAPA), which combines deterministic verifier-based process rewards, step-level action comparisons through forked replay, and a two-level advantage formulation without requiring an additional critic or reward model. Experiments on MIMIC-IV-related tasks, MedAgentBench, EHRSHOT, and chained benchmarks demonstrate generally consistent improvements over GRPO and other RL baselines, supported by several ablation and mechanism analyses. Overall, the problem is practically relevant and the empirical results are promising.

Critical strengths
  1. The research problem is important and practically relevant. Assigning appropriate training signals to intermediate decisions in long-horizon agent trajectories is an important challenge in reinforcement learning-based agent optimization. Compared with optimizing solely based on final answers, distinguishing the contributions of retrieval, memory updates, and stopping decisions could improve learning efficiency in long-horizon tasks. The paper's focus on EHR tasks requiring multiple queries and evidence integration is therefore well motivated.
  2. Incorporating structured EHR information into process reward design has practical value. The paper employs deterministic verifiers to assess evidence grounding, temporal validity, unit consistency, missingness, and memory operations. Compared with relying entirely on learned process reward models, this approach provides more interpretable supervision based on structured records and explicit task rules, while reducing dependence on additional reward models. The authors also conduct a verifier audit to evaluate the reliability of different verifier categories.
  3. The experimental evaluation is relatively comprehensive and includes valuable controlled comparisons. Beyond GRPO, the paper compares against VinePPO, GiGPO, Tree-GRPO, and other critic-based and critic-free baselines. The authors conduct factorial ablations to examine the effects of process rewards and step-credit regimes, and compare forked replay with additional independent rollouts under a matched sampled-token budget. Experiments varying memory capacity and decision depth, as well as evaluations on a second model backbone and held-out datasets, further help assess the method's applicability.
Critical weaknesses

The entire problem formulation relies on a highly specific memory management architecture: tool outputs are visible only in the next observation, the action history does not retain clinical values or reasoning text, and the agent must explicitly save information through memory operations, subject to a strict limit on the number of memory items (L=8 in the main experiments). However, this fixed-slot memory management mechanism, which requires explicit information deletion, is not representative of common context management practices in modern agents. Modern agents (e.g., claude code/codex) often rely on longer context windows, automatic context compaction, external persistent storage, and on-demand retrieval, rather than a very small structured memory that requires explicit discard operations.

This concern is particularly important because several core examples and verifier checks directly focus on incorrect memory discard decisions. Although the authors provide ablations varying memory capacity (L), these experiments retain the same fixed-slot memory architecture and therefore cannot establish whether VAPA's benefits persist under different memory management mechanisms.

I suggest that the authors include at least one baseline that compares the bounded-memory design with the common agent framework (e.g., pi agent), such as an agent using long-context processing with automatic compaction. If there are experiments shown that the bounded-memory design in this paper is more suitable for EHR tasks, then the whole assumption should be meaningful. The authors should more clearly justify why fixed item-level memory and one-turn tool visibility are appropriate abstractions for EHR agents, and explain their practical advantages over alternative context management approaches.

Without such evidence, the paper's conclusions are primarily supported within the specific bounded-memory environment constructed by the authors, while their generalizability to broader EHR agents and long-horizon tool-using agents remains unclear.

Questions for authors

See the weaknesses.

2 comments

We thank Reviewer BLQd for questioning whether our bounded-memory design is representative of EHR agents. We explain how it relates to common context management and add GRPO and VAPA under automatic compaction.

  • Bounded memory. We agree that our memory is stricter than in general-purpose agents. It shares two of the practices the reviewer lists, since the patient record serves as external storage queried on demand and memory is the small working set kept between queries. One-turn visibility makes each kept value an explicit write with an evidence pointer that grounded answers and the verifiers check. Section 2 now gives this justification.
  • Compaction. As an alternative design, we train GRPO and VAPA in a harness that keeps full tool returns in the same 32,768-token context, summarizes older content when it fills, and has no memory actions. The backbone, task tools, action budget, and actor-token budget match the main runs, and VAPA uses the predicates that apply without memory actions (Appendix C.3).
Context managementMethodGrounded EM (%)MedAgentBench-Query success (%)
Bounded memory, L = 8GRPO70.2132.53
Bounded memory, L = 8VAPA78.0342.67
Full tool returns + compactionGRPO72.1235.07
Full tool returns + compactionVAPA79.0443.60

32,768-token window, five seeds at the full token budget. Bounded rows are Table 21 at L = 8.

VAPA improves over GRPO under both designs, so the gain is not specific to bounded memory. Both methods score somewhat higher in this harness, while bounded memory makes retention explicit and provenance-linked. The comparison is in Appendix C.3.

Dear Reviewer BLQd,

Thanks again for the review. Comparing against automatic compaction was a good test of our memory design, so we ran it: GRPO and VAPA in that setting are now in Appendix C.3, and Section 2 explains why we chose bounded memory for EHR agents.

Please let us know if any part of your concern remains. We'd be glad to follow up before the discussion closes.

Best regards,

Authors of Paper 7868

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.