acceptodds
Under review as a conference paper at ICLR 2027

Mem-C: Scaling and Allocating Credit Beyond Dense Rewards in Long-Horizon Agents

Abstract

Long-horizon language agents face severe temporal credit-assignment challenges, as the consequences of intermediate decisions may only emerge many steps later. Dense process supervision and tree-based training alleviate this issue by providing additional downstream feedback, but existing work largely treats credit assignment as a one-dimensional notion of reward density. This overlooks a key fact: the same credit budget can be distributed differently across decisions, and additional supervision may have very different marginal value depending on where it is assigned. We therefore ask: how does agent performance scale with credit, and how should a fixed credit budget be allocated across long-horizon decisions? To answer this question, we introduce **Credit Assignment Geometry**, which characterizes step-level credit along three dimensions: ① ***Q**uantity*, ② ***H**eterogeneity*, and ③ ***U**tility*. We find that increasing credit quantity can yield diminishing returns, while allocations with comparable quantity and coverage can yield substantially different performance due to heterogeneous credit utility. Motivated by these findings, we propose **Mem-C**, which uses branch-outcome dispersion to identify high-utility decisions for importance-aware branch allocation. Under matched credit budgets, **Mem-C** outperforms the evaluated baselines on all three transfer benchmarks, including **12%↑** on HotpotQA. Code is available at [Mem-C](https://anonymous.4open.science/r/Mem-C-543B/).

Primary area:
reinforcement learning
Keywords:
Credit Assignment Geometry, Dense Process Supervision, Long-Horizon Language Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ATdLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies temporal credit assignment in RL training of long-horizon memory agents. It argues that prior work treats credit assignment as a one-dimensional question of reward density, and proposes "Credit Assignment Geometry," which describes step-level credit along three dimensions: Quantity (Q), Heterogeneity/Coverage (H), and Utility (U). Building on this view, the authors propose Mem-C, a tree-based RL method whose branch selection combines a retrieval-coverage score with a utility proxy Û, defined as the mean pairwise difference in outcomes across a node’s child branches. Expansion budget is thus steered toward decisions whose continuations lead to more divergent outcomes.

Mem-C is validated on LoCoMo (59.41 F1, vs. 58.65 for Mem-T) and on three transfer benchmarks (HotpotQA, LongMemEval, NarrativeQA), where it outperforms all baselines. The paper also includes three analysis experiments, varying search budget, the coverage parameter, and the utility weight βU, which are presented as evidence for diminishing returns and heterogeneous credit utility.

Critical strengths
  1. The paper reframes credit assignment in tree-based agentic RL as a budget allocation problem, pointing out that under the same rollout budget, where branches are placed matters as much as how dense the reward is.

  2. The utility proxy reuses already-scored branch outcomes, so it adds no extra rollouts and can be plugged into existing tree-based RL pipelines with little effort.

Critical weaknesses
  1. The paper is motivated by credit assignment over long-horizon memory operations, yet Appendix A.2 shows that RL only optimizes a retrieval policy over a pre-built memory store, with memory construction kept outside the RL loop. The construction and revision decisions emphasized in the motivation are therefore never trained.

  2. The paper claims to address long-horizon credit assignment, but Fig. 8 shows a median inference trace length of only 2 steps, with two-step traces making up over 80% of records. The training configuration also caps retrieval at 6 turns (Table 6). This setting does not convincingly support the "long-horizon" claim.

  3. The central question covers both how learning scales with credit and how credit should be allocated. Mem-C only addresses allocation under a fixed budget, and the scaling question receives no methodological answer.

  4. The paper does not explain why the mean pairwise absolute difference of branch outcomes is used as the utility proxy rather than variance or another dispersion measure.

  5. No justification is given for sampling a random anchor node at the start of each expansion round.

  6. Figs. 1 and 2 repeatedly state that "high-entropy steps need more reward," implying an entropy-based method. The method described in the main text instead relies on branch-outcome dispersion and never involves entropy.

  7. Fig. 2 includes a Retry Gate module that is not described anywhere in the method section. The paper does not state whether it is used in the main results.

Questions for authors
  1. Mem-C improves over Mem-T by only 0.76 F1 on in-domain LoCoMo but by 12.82 F1 on out-of-domain HotpotQA. What explains this gap? Is the HotpotQA gain attributable to utility-aware allocation, or to other factors such as the SFT stage, the Retry Gate, or differences in evaluation settings?

  2. Outcome dispersion across child branches also reflects sampling randomness (temperature 1.0) and reward noise, not only the decision at node v. How do the authors ensure that a high Û indicates a consequential decision rather than a noisy downstream subtree?

  3. Since Û is zero for nodes with fewer than two scored branches, the selector appears biased toward already-expanded nodes. How does Mem-C avoid this exploitation bias and still discover new high-utility decisions?

11 comments
AuthorsOpenReview ↗

Response to Reviewer ATdL on Performance Disparities, Utility Estimation, and Exploration Strategy

Question 1 & Weakness 7:

Question 1: Mem-C improves over Mem-T by only 0.76 F1 on in-domain LoCoMo but by 12.82 F1 on out-of-domain HotpotQA. What explains this gap? Is the HotpotQA gain attributable to utility-aware allocation, or to other factors such as the SFT stage, the Retry Gate, or differences in evaluation settings?

Weakness 7: Fig. 2 includes a Retry Gate module that is not described anywhere in the method section. The paper does not state whether it is used in the main results.

Authors’ Response:

  • Thank you for raising the concerns.

    • We will clarify the role of the Retry Gate and isolate the contribution of utility-aware allocation to explain the substantial performance gain on HotpotQA.

    • 1. Clarification of the Retry Gate.

      • The Retry Gate is an inference-time conditional retrieval mechanism that determines whether an additional retrieval pass is needed when the initial retrieval results are insufficient or uncertain.
    • 2. Ablation of utility-aware allocation and the Retry Gate.

      • To isolate their respective contributions, we conduct a 2 × 2 controlled ablation on HotpotQA, independently varying the utility coefficient and the Retry Gate.

      • Specifically, we compare coverage-only allocation βU=0\beta_U=0 with utility-aware allocation βU=1\beta_U=1, under both enabled and disabled Retry Gate settings.

        • All four configurations use the same initial policy checkpoint, training protocol, memory store, test examples, and evaluation settings.

        • The results are presented in Table R3-1.

        • Table R3-1: Controlled Ablation of Utility-aware Allocation and Retry Gate on HotpotQA.

        • Utility-aware AllocationRetry GateHotpotQA F1
          ✗✗65.6
          ✓✗77.5
          ✗✓67.0
          ✓✓79.2
          • Two observations can be drawn from these results.
          • Contribution of utility-aware allocation.
            • Enabling utility-aware allocation improves F1 by 11.9 points without the Retry Gate and 12.2 points with the Retry Gate.
              • This consistent improvement indicates that the benefit of utility-aware credit allocation is not depend on the Retry Gate.
          • Contribution of the Retry Gate.
            • Enabling the Retry Gate yields only 1.4 and 1.7 additional F1 points under the utility-disabled and utility-enabled settings.
              • Its contribution is substantially smaller than that of utility-aware allocation.
        • These results indicate that utility-aware credit allocation accounts for the dominant part of the observed HotpotQA improvement, whereas the Retry Gate provides a relatively small additional benefit.

          • We will add these analyses in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer ATdL on Performance Disparities, Utility Estimation, and Exploration Strategy

Question 2 & Weakness 4:

Outcome dispersion across child branches also reflects sampling randomness (temperature 1.0) and reward noise, not only the decision at node v. How do the authors ensure that a high Û indicates a consequential decision rather than a noisy downstream subtree?

The paper does not explain why the mean pairwise absolute difference of branch outcomes is used as the utility proxy rather than variance or another dispersion measure.

Authors’ Response:

  • 1. Estimator definition and rationale.

    • Our implementation first computes the mean observed return within each scored child branch:
      • $ V_i(v)=\frac{1}{|\mathcal L_i^{\mathrm{scored}}|} \sum_{\ell\in\mathcal L_i^{\mathrm{scored}}}R_\ell. $
    • For B≥2B\geq2 scored child branches, the utility estimate is defined as
      • $ \hat U(v)=\frac{2}{B(B-1)} \sum_{i<j}|V_i(v)-V_j(v)|. $
    • When fewer than two scored branches are available, the estimate is marked as unavailable and contributes zero to the selection score. It is updated as additional scored outcomes become available.
    • We choose the mean pairwise absolute difference for three reasons:
      • Interpretability: It directly measures the average observed return difference between two distinct child branches, providing a simple indicator of outcome sensitivity.
      • Linear reward scale: Unlike variance, which uses squared differences, the estimator remains in the same units as the returns, making its combination with the coverage score through βU\beta_U straightforward.
      • Branch-count normalization: Averaging across branch pairs avoids the mechanical increase associated with summing pairwise differences as more branches become available.
    • Variance is also a reasonable alternative. For exactly two branch values, population variance equals U^(v)2/4\hat U(v)^2/4, so both estimators produce the same ranking. With more branches, their rankings may differ because variance places greater weight on extreme values. We do not claim that our estimator is inherently superior to variance or other dispersion measures.
  • 2. Sampling randomness and interpretation of utility.

    • We agree that outcome dispersion reflects not only differences between child decisions but also downstream sampling randomness (temperature 1.0) and reward noise.
    • Our estimator partially mitigates individual-outcome fluctuations by averaging all available scored descendant returns within each child branch, rather than comparing individual rollouts directly.
      • Since child branches share the same decision prefix, the resulting statistic measures differences in observed outcomes among alternative continuations from that prefix.
    • However, averaging does not eliminate sampling noise or establish that the observed differences are caused by the branching decisions.
      • In particular, finite-sample fluctuations and downstream stochasticity may still produce high utility estimates.
    • Therefore, we interpret a high U^(v)\hat U(v) as evidence of observed outcome sensitivity, rather than a guaranteed indicator of causal decision importance or marginal learning benefit.
      • Its role is to provide a heuristic signal for prioritizing further branching.
  • 3. Empirical evidence and limitations.

    • The supplementary utility-weight sweep provides a diagnostic of how the estimator affects node selection:

      • βU\beta_UMean selected-node U^\hat UMean logged training reward
        0.00.03340.1557
        0.50.06110.1258
        1.00.07620.1363
        1.50.07790.1317
        2.00.07720.1247
      • These statistics cover the common first 29 updates and all three expansion rounds, including the initial round with zero utility estimates. Increasing the utility weight generally leads to the selection of nodes with higher observed utility, confirming that the estimator influences expansion decisions.

        • We will revise the manuscript to clarify both the motivation for the mean pairwise absolute difference and its limitations, consistently describing U^\hat U as an observed outcome-sensitivity heuristic rather than a noise-free measure of causal decision importance.
AuthorsOpenReview ↗

Response to Reviewer ATdL on Performance Disparities, Utility Estimation, and Exploration Strategy

Question 3:

Since Û is zero for nodes with fewer than two scored branches, the selector appears biased toward already-expanded nodes. How does Mem-C avoid this exploitation bias and still discover new high-utility decisions?

Authors’ Response:

Insufficient utility evidence does not exclude a node from selection. The implementation provides three mechanisms support exploration:

Random anchor.

  • Each expansion round first samples uniformly from the full candidate set—the root and all nonterminal descendants—regardless of utility readiness.
    • One of the nn slots therefore bypasses utility ranking: half the slots in Exp-C (n=2n=2).

Coverage-based selection.

  • The remaining slots use Ch(v,S)+βUU^(v)C_h(v,S)+\beta_U\hat U(v). A node without a utility estimate can still compete through coverage, including retrieved-evidence novelty.
    • Candidates are selected without replacement until the pool is exhausted.

Estimate refresh.

  • Newly generated leaves are scored, and utilities are recomputed before the next round, allowing exploration to establish utility estimates for previously unassessed nodes.

Empirical check. In a frozen-policy audit with eight validation questions and three sampling seeds, utility-unready nodes continue receiving expansion slots:

  • Expansion roundSelection slots assigned to utility-unready nodes
    1100.00% ± 0.00%
    246.88% ± 8.27%
    338.54% ± 3.61%
  • Utility-unready denotes fewer than two scored child branches. Values are mean ± sample SD across three sampling seeds.

    • These measurements demonstrate continued expansion of nodes whose utility cannot yet be estimated.
  • We sincerely appreciate your constructive feedback and positive assessment of the novelty, performance, and contributions of our work.

    • We understand that your remaining concerns primarily focus on the need for additional experimental validation and clarification of specific details,
      • rather than questioning the core novelty or effectiveness of our approach.
      • To address these concerns, we have conducted supplementary experiments and provided more detailed explanations.
        • We hope that these additional results and clarifications have adequately addressed your concerns.
          • If so, we would be sincerely grateful if you could reconsider your overall score.

Weakness 1:

The paper is motivated by credit assignment over long-horizon memory operations, yet Appendix A.2 shows that RL only optimizes a retrieval policy over a pre-built memory store, with memory construction kept outside the RL loop. The construction and revision decisions emphasized in the motivation are therefore never trained.

Authors’ Response:

  • Mem-C instantiates credit allocation over the retrieval policy of a memory agent.

    • Memory construction and revision prepare the store outside the RL loop; the optimized decisions are summarized below.

    • OperationImplementationRole during RL
      Memory constructionForm memories from dialogueMemory preparation outside the RL loop
      Memory revisionUpdate stored memory entriesMemory preparation outside the RL loop
      Memory-source selectionChoose facts, experiences, personas, summaries, or raw turnsRL-optimized policy decision
      Query formulationGenerate tool arguments and reformulate queriesRL-optimized policy decision
      Stopping and answeringEmit finish with the final answerRL-optimized policy decision
    • A fixed memory store still leaves consequential sequential choices: the selected source and query determine the evidence available to subsequent decisions.

      • Mem-C uses observed continuation outcomes to prioritize branching within this retrieval sequence.
        • Keeping the store fixed provides a common environment for these comparisons.
    • We will specify this action space consistently in the introduction and figures, distinguishing memory preparation from the retrieval decisions optimized by RL.


Reviewer_ibhrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In tasks where an AI accesses information from memory and takes several steps before answering, it can be difficult to know which earlier decisions contributed to success or failure. This paper studies how a limited AI training budget is used more effectively. The proposed method, Mem-C, explores alternative ways and compares their outcomes. It focuses on decisions where the paths yield different rewards, as these may provide more useful training opportunities. The core idea of the paper is that where you spend the training budget matters, together with how much you spend when dealing with long horizon tasks.

Critical strengths

the article explores an important problem: How can an AI learn more effectively with limited training? It proposes a clear framework for understanding training feedback by dividing the challenge into three questions: Amount: How much feedback does the AI receive? Distribution: Is that feedback spread across many decisions or concentrated on a few? Usefulness: How much does that feedback help the AI learn to make better decisions? The method is not overwhelmingly complex: It uses outcomes from branches already explored to decide where to spend additional training effort.

Critical weaknesses

the paper proposes a reasonable pattern, giving the AI more training feedback improves performance, but each additional increase helps less than the previous one, but does not show enough evidence that the pattern actually holds. It is unclear which factor causes the improvements. The paper distinguishes the amount, distribution, and usefulness of supervision, but the experiments do not adequately distinguish them. The same number of search branches does not guarantee the same amount of feedback or computation. The experiments supporting the main idea are too limited. The conclusion says the method consistently exceeds random allocation. However, the appendix acknowledges that its experiments do not establish this, which one

Questions for authors

Section 3.2 describes the scaling curve as illustrative, while the introduction and conclusion present diminishing returns as a finding. Which experimental results establish this relationship? Which experiment supports the conclusion that Mem-C outperforms random allocation? How many evaluation examples and training runs support each result in tables 1 and 2? In the utility-aware versus coverage-only comparisons, what were the actual credit quantities, coverage, and total computational costs?

5 comments
AuthorsOpenReview ↗

[1/2] Response to Reviewer ibhr on Diminishing Returns, Credit Allocation, and Computational Cost

Question 1:

Section 3.2 describes the scaling curve as illustrative, while the introduction and conclusion present diminishing returns as a finding. Which experimental results establish this relationship? Which experiment supports the conclusion that Mem-C outperforms random allocation? How many evaluation examples and training runs support each result in tables 1 and 2? In the utility-aware versus coverage-only comparisons, what were the actual credit quantities, coverage, and total computational costs?

Authors’ Response:

Thank you for these questions. We clarify the interpretation of the insight, the supporting comparisons, and the experimental accounting below.

  • 1. Interpretation of the scaling insight.

    • Insight I concerns the marginal benefit of additional search in the tested settings.
      • For example, in Table 3, increasing expansion width at fixed m=4m=4 raises LoCoMo F1 from 58.80 to 59.45 to 59.74 as QsearchQ_{search} increases from 12 to 16 to 20: the successive gains are +0.65 and +0.29 points. Further widening to 24 reduces F1 to 59.41.
      • These observations motivate the insight that
        • additional search can provide diminishing or negative returns. Equation (3) illustrates a possible saturating relationship; it is not a fitted or universally established scaling law.
  • 2. Evidence concerning random allocation.

    • The relevant benchmark comparison is Mem-C versus the tree-based Mem-T baseline in Tables 1–2.
      • The manuscript reports gains of +0.76 F1 on LoCoMo, +12.82 F1 on HotpotQA, +2.23 accuracy points on LongMemEval, and +4.88 F1 on NarrativeQA.
      • The implemented random selector uniformly samples expandable nodes with replacement. However, the utility-weight sweep compares against coverage-only selection (βU=0\beta_U=0), not that random selector.
  • 3. Evaluation examples and training runs.

    • Appendix A.1 specifies 152/81/1,307 training/validation/test questions for LoCoMo, with the 1,307 test questions drawn from eight held-out conversations.
      • The identifiable utility-aware checkpoint is associated with seed 42; evaluations on different benchmarks, evaluation restarts, and checkpoints from the same training trajectory are not independent training runs.
  • 4. Credit quantity, coverage, and computational cost.

    • The utility-aware and utility-disabled comparisons use the same nominal search configuration (m,n,l,k)=(2,2,3,1)(m,n,l,k)=(2,2,3,1) and coverage-preference parameter h=0.5h=0.5.

      • These settings specify the intervention; QsearchQ_{search} and hh are not measurements of realized credit quantity QQ and credit-distribution entropy HH.

      • Additional frozen-policy diagnostics. We compare the three selectors using the same Mem-T checkpoint, memory store, eight validation questions, and three sampling seeds, fixing Qsearch=14Q_{\text{search}}=14, h=0.5h=0.5, and the scoring schedule. No policy updates are performed.

        Realized terminal-feedback statistics. Each condition generates 14 scored terminal outcomes and retains two trajectories per question. For each retained path, cˉ\bar c averages the number of scored terminal descendants per decision; HH measures the normalized entropy of these counts.

        SelectorAvailable cˉ\bar cAvailable HHRetained cˉ\bar cRetained$H$
        Random2.311 ± 0.2350.839 ± 0.0211.0001.000
        Coverage-only2.789 ± 0.1270.829 ± 0.0251.0001.000
        Utility-aware (βU=1\beta_U=1)2.688 ± 0.2650.834 ± 0.0331.0001.000

        Available and retained statistics are measured before and after pruning, respectively. These count terminal-feedback dependencies, rather than independent rewards or all process-supervision signals.

AuthorsOpenReview ↗

[2/2] Response to Reviewer ibhr on Diminishing Returns, Credit Allocation, and Computational Cost

      • Measured rollout cost. We additionally report actual token usage and runtime alongside the nominal budget.

        SelectorCumulative Input Tokens / QuestionGenerated Tokens / QuestionAmortized Time (s/question)GPU-hours
        Random159,792 ± 29,41210,781 ± 1,454132.90 ± 15.340.886
        Coverage-only152,020 ± 13,8858,857 ± 1,271118.02 ± 13.950.787
        Utility-aware166,850 ± 51,5179,883 ± 2,813137.69 ± 36.360.918

        Values are mean ± sample SD across three sampling-seed means. Each worker uses one RTX A6000; GPU-hours sum the three runs. Runtime includes generation, retrieval, and scoring, excluding loading, warm-up, and optimization.

        On 144 matched real tree snapshots, utility-aware selection takes 0.1100 ms/call, with utility refresh requiring 0.0299 ms/call, measured separately from generation and reward evaluation.

      • Training-stage cost. Separately, we summarize the common first 29 updates from the archived training runs:

        • MeasurementUtility-disabled control (βU=0\beta_U=0)Mem-C (βU=1\beta_U=1)
          Nominal search budget QsearchQ_{search}1414
          Coverage-preference parameter hh0.50.5
          Training seed / compared updates42 / 2942 / 29
          Logged training-stage time (s/update)836.03863.82
          29-update logged-stage time (h)6.7356.959
          Estimated configured-GPU-hours*53.8855.67
      • GPU-hours are calculated using eight configured GPUs.

        • Logged time includes generation, reference scoring, advantage computation, and actor update; it excludes initialization, memory construction, SFT, validation, checkpoint saving, and unlogged work. The measured increase is 3.32%.
      • The fixed-length checkpoint decoding measurements additionally show comparable inference cost: for 512/128 input/output tokens, Mem-T and Mem-C take 5.225 ± 0.159 s and 5.173 ± 0.068 s; both peak at 7.667 GiB allocated memory.

  • We thank the reviewer for the constructive feedback.

    • We believe the concerns mainly concern additional experimental analysis and clarification,
      • rather than fundamental limitations of our approach.
      • We have carefully addressed these points with supplementary experiments and analyses, and sincerely hope the reviewer will reconsider the overall score in light of these revisions.
AuthorsOpenReview ↗

Response to Reviewer ibhr on Diminishing Returns, Credit Allocation, and Computational Cost

Weakness 1:

the paper proposes a reasonable pattern, giving the AI more training feedback improves performance, but each additional increase helps less than the previous one, but does not show enough evidence that the pattern actually holds. It is unclear which factor causes the improvements. The paper distinguishes the amount, distribution, and usefulness of supervision, but the experiments do not adequately distinguish them. The same number of search branches does not guarantee the same amount of feedback or computation. The experiments supporting the main idea are too limited. The conclusion says the method consistently exceeds random allocation. However, the appendix acknowledges that its experiments do not establish this, which one

Authors’ Response:

We thank the reviewer for emphasizing the distinction between search budget, credit allocation, and computation.

  • Budget response.

    • Exp-A examines how performance changes with nominal search budget.

      • Table 3 shows task-dependent, non-monotonic returns: at fixed m=4m=4, LoCoMo peaks at Qsearch=20Q_{\text{search}}=20 and declines at 24. These observations motivate selective allocation; Eq. (3) provides an illustrative model of diminishing returns.

      • Config.mmnnQsearchQ_{\mathrm{search}}LoCoMo (F1↑)HotpotQA (F1↑)LongMemEval (Acc.↑)
        Increasing tree count (n=1n=1)
        11259.0769.3060.00
        21459.1675.9760.00
        31658.5763.9760.00
        41859.3663.9765.00
        511058.0763.9760.00
        Increasing expansion width (m=4m=4)
        421258.8070.6355.00
        431659.4570.6360.00
        442059.7469.3055.00
        452459.4167.0560.00
  • Allocation comparison.

    • Exp-C varies βU\beta_U while fixing (m,n,l,k)=(2,2,3,1)(m,n,l,k)=(2,2,3,1) and h=0.5h=0.5.
      • This compares utility weighting against the coverage-only selector under the same nominal search configuration. We distinguish these controlled settings from realized credit quantity and coverage.
  • Measured computational cost.

    • We additionally quantify runtime rather than infer it from branch counts.

      • Over the common first 29 updates, Mem-C (βU=1\beta_U=1) takes 863.82 s/update, compared with 836.03 s/update for the utility-disabled, scoring-matched control: 3.32% additional recorded-stage time.

        • This measurement covers generation, reference scoring, advantage computation, and actor updates.

        • MethodTraining Time (s/update)29-step Time (h)Estimated GPU Hours
          Utility-disabled Control836.036.73553.88
          Mem-C ((\beta=1))863.826.95955.67
          Overhead+3.32%+3.33%+3.32%
  • Comparison scope.

    • Tables 1–2 report benchmark improvements over Mem-T, whereas the utility-weight sweep compares against coverage-only selection.
Reviewer_b6poOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Mem-C, a method designed to improve existing credit assignment approaches by identifying high-utility decisions and enabling importance-aware branch allocation. Under matched credit budgets, Mem-C outperforms the evaluated baseline methods across all three transfer benchmarks, demonstrating its effectiveness in allocating credit more strategically and improving performance across transfer tasks.

Critical strengths

The proposed method is presented clearly, intuitively, and in a well-structured manner, making the overall framework, its key components, and their interactions easy to follow and understand. The paper coherently explains the methodological design and effectively motivates the role of each component within the broader approach. This clarity facilitates understanding of the underlying rationale and helps readers appreciate how the different components work together to achieve the intended objectives. Overall, the well-organized presentation enhances the accessibility of the proposed method and makes the paper easier to follow.

Critical weaknesses
  1. Some of the main results demonstrate only marginal improvements over baseline methods, raising questions about the statistical significance and practical relevance of the claimed advantages. A more rigorous statistical analysis, including confidence intervals and significance tests, would help establish the reliability and consistency of these improvements across different tasks. Furthermore, the paper would benefit from more detailed case studies and qualitative analyses of credit assignment quality. For example, visualizing the credit assigned to individual components and comparing these allocations with those produced by alternative methods would provide deeper insights into the effectiveness and interpretability of the proposed approach.

  2. The paper would benefit from a more comprehensive analysis of the computational costs associated with Mem-C compared with competing methods. Although the proposed framework demonstrates promising performance, the additional overhead introduced may affect its practical applicability. This analysis is particularly important because some reported performance improvements over the baselines are marginal, making it necessary to establish whether these gains justify the additional computational resources. The authors should therefore report and compare metrics such as training time, GPU hours, and inference cost across Mem-C and the baseline methods. If the performance gains are modest relative to the additional computational costs, the practical advantages of Mem-C may be less substantial than the results suggest.

Questions for authors

Please refer to the Weaknesses section, as they are also my concerns and questions for the authors. Any additional clarifications would be welcome.

3 comments
AuthorsOpenReview ↗

Response to Reviewer b6po on statistical reliability, credit quality, and computational cost

Weakness 1:

Some of the main results demonstrate only marginal improvements over baseline methods, raising questions about the statistical significance and practical relevance of the claimed advantages. A more rigorous statistical analysis, including confidence intervals and significance tests, would help establish the reliability and consistency of these improvements across different tasks. Furthermore, the paper would benefit from more detailed case studies and qualitative analyses of credit assignment quality. For example, visualizing the credit assigned to individual components and comparing these allocations with those produced by alternative methods would provide deeper insights into the effectiveness and interpretability of the proposed approach.

Authors’ Response:

We thank the reviewer for the constructive suggestions.

  • Statistical reliability.

    • While some individual improvements are modest, Mem-C achieves consistently competitive performance across multiple benchmarks, demonstrating its effectiveness across diverse settings rather than on a single dataset.
  • Qualitative analysis. We will extend the analyses in Figures 5 and 15 with three additional paired case studies comparing Mem-C against the historical baseline:

  • CaseGround-truth EvidenceHistorical Comparison (Exp-A)Mem-C Trajectory and AnswerDecision-level Diagnostic
    1. Recovering a Resumed Activity (conv-49_72) Question: "Which activity did Sam resume in December 2023 after a long time?"On December 17, Sam mentioned that he used to love hiking but had not done it for a while (D20:6). On December 31, he reported a recent hiking trip (D22:1). Gold: hiking.Six raw-turn searches followed by a fallback answer: "yoga" (incorrect; 7 trace entries).Two fact searches → two raw-turn searches → finish: "hiking" (correct; 5 trace entries).The trace demonstrates a useful retrieval-source transition rather than repeatedly broadening similar queries. The shift from incomplete fact retrieval to raw-turn evidence is a candidate decision for credit-allocation analysis.
    2. Counting Distinct Events Across Sessions (conv-42_62) Question: "How many letters has Joanna recieved?"A company rejection letter appears in Session 14 (D14:1), while a reader's letter responding to her blog appears in Session 18 (D18:5). Gold: two.Six fact searches repeatedly sought an explicitly stated total, followed by a fallback answer: "1 letter" (incorrect; 7 trace entries).The first fact search retrieved one letter; a second search broadened the query to other received letters → finish: "two" (correct; 3 trace entries).The consequential decision is to search for additional distinct events rather than continue looking for an explicitly stated numerical total. This case distinguishes evidence aggregation from redundant searches for a numeral.
    3. Correct Evidence, Inconsistent Final Decision (conv-47_3) Question: "Do both James and John have pets?"James refers to his dogs (D1:12), whereas John says that he wants a pet but is "not there yet" (D2:18). Gold: no.Five fact searches → one raw-turn search → fallback. The final answer correctly states that John does not have pets and concludes "No" (semantically correct; 7 trace entries).Separate fact searches for James and John → finish: "Yes, James has pets; John does not." (incorrect; 3 trace entries).The failure occurs during the conversion of correctly retrieved ownership facts into a final yes/no decision. The leading "Yes" contradicts the conjunction in the question. This case demonstrates that shorter retrieval traces do not necessarily produce better decisions
AuthorsOpenReview ↗

Response to Reviewer b6po on statistical reliability, credit quality, and computational cost

Weakness 2:

The paper would benefit from a more comprehensive analysis of the computational costs associated with Mem-C compared with competing methods. Although the proposed framework demonstrates promising performance, the additional overhead introduced may affect its practical applicability. This analysis is particularly important because some reported performance improvements over the baselines are marginal, making it necessary to establish whether these gains justify the additional computational resources. The authors should therefore report and compare metrics such as training time, GPU hours, and inference cost across Mem-C and the baseline methods. If the performance gains are modest relative to the additional computational costs, the practical advantages of Mem-C may be less substantial than the results suggest.

Authors’ Response:

We thank the reviewer for highlighting the importance of computational efficiency.

  • We conduct additional measurements to evaluate Mem-C's training and inference costs.

    • Training efficiency. We compare Mem-C (βU=1)(\beta_U=1) with a utility-disabled, scoring-matched control under the same nominal search budget (Qsearch=14)(Q_{\text{search}}=14) over the first 29 optimization steps.

      • MethodTraining Time (s/update)29-step Time (h)Estimated GPU Hours
        Utility-disabled Control836.036.73553.88
        Mem-C ((\beta=1))863.826.95955.67
        Overhead+3.32%+3.33%+3.32%
    • Inference efficiency. We further benchmark Mem-T and Mem-C checkpoints under identical hardware and decoding configurations, using fixed input and output lengths.

      • CheckpointInput / Output TokensLatency (s) ↓Throughput (tokens/s) ↑Peak GPU Memory (GiB) ↓
        Mem-T512 / 1285.225 ± 0.15924.507.667
        Mem-C512 / 1285.173 ± 0.06824.747.667
        Mem-T2048 / 1285.306 ± 0.12524.127.987
        Mem-C2048 / 1285.258 ± 0.11024.357.987
    • Overall, Mem-C introduces only 3.32% additional logged training overhead, while achieving comparable checkpoint-level inference efficiency, with nearly identical latency and GPU memory consumption.

Dear Reviewer b6po,

We sincerely appreciate your time and effort in providing such thoughtful reviews and insightful comments.

Could you please kindly let us know if our rebuttal has addressed your concerns?

If you have any further questions regarding our work, we would be delighted to address them.

If our responses have addressed your concerns, we would greatly appreciate it if you would consider whether the paper merits a higher overall rating.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.