acceptodds
Under review as a conference paper at ICLR 2027

Dual-Grained Agent Memory and Shapley Context Attribution for Multimodal Agentic Learner

Abstract

External memory allows frozen multimodal models to bypass costly parameter up- dates by leveraging labeled training data at inference. Designing effective memory, however, requires balancing the specific visual and logical grounding of individual cases with the need for generalized, transferable procedures. Furthermore, jointly retrieving multiple memories complicates selection by frequently introducing redun- dant or contradictory context. To address these challenges, we introduce DG-Mem, an offline memory framework that extracts dual-grained representations—instance- grounded exemplars and category-level schemas—directly from a frozen model’s training rollouts. Per-problem reflections are dynamically clustered by an incre- mental categorizer and synthesized into shared procedural advice. To optimize test-time ranking and resolve retrieval conflicts, a secondary offline pass evaluates subsets of these rules to assign Shapley-based utilities. Evaluated on held-out parti- tions of MathVista, MMMU, and MMMU-Pro, DG-Mem consistently improves accuracy over retrieval-free prompting across both open-weight and proprietary backbones. The framework maintains this advantage across repeated trials and outperforms tool-free XSkill under randomized splits. Applying DG-Mem to text- only mathematics yields similar gains: memory constructed from DAPO problems improves pooled AIME and HMMT accuracy, surpassing TFGRPO. Ablations confirm that the dual memory grains function complementarily, and Shapley utility weighting successfully provides further incremental gains. Ultimately, DG-Mem validates offline memory synthesis as a highly effective, practical adaptation strat- egy for frozen models, establishing a strong foundation for future advancements in rule-level context attribution.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
agentic memory, attribution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XJr9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a test-time method that augment frozen multimodal models based on an external memory. The external memory is constructed by extracting reasoning rules and problem-related exemplars and used to boost the performance at the inference time with a Shapley-based utility estimation. The experimental results support their claims in the main text.

Critical strengths
  1. They propose an interesting method, which considers general reasoning rules and specific exemplars to allow the multimodal models to leverage both general knowledge and concrete reasoning experiences.
  2. They point out that relying solely on similarity between the rules and problems are not enough, and they design a detailed method to solve this.
Critical weaknesses
  1. The novelty is not enough. They design a method for test-time inference, where they introduce a constructed external memory and design a detailed utility estimation. From my end, the contribution is combining an external memory with a detailed utility estimation, which is not novel for ICLR conf level.

  2. Considering the experiments are conducted only under testtime, maybe the more comprehensive experiments could be performed and needed. The current setup shows that the baselines are only three methods and they just used 3 datasets. It'd be better if they include more experiments.

Questions for authors

Could you explain why there are drops (red results in Table 2)?

Reviewer_QTveOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DG-Mem, a training-free multimodal reasoning framework that constructs instance-level exemplars and category-level procedural memories from model rollouts, and employs Shapley-based context attribution to improve memory retrieval.

Critical strengths
  1. The paper addresses the practical challenge of balancing instance-specific knowledge and transferable procedural guidance in external memory, while considering redundancy and interactions among retrieved memories.
  2. DG-Mem demonstrates improvements across MathVista, MMMU, and MMMU-Pro with multiple open-weight and proprietary models. Additional comparisons with ViLoMem, XSkill, and TFGRPO, together with repeated evaluations, provide useful evidence of its effectiveness.
  3. The dual-grained memory construction is intuitive, and the ablations provide evidence that exemplar and schema memories can complement each other. The paper also explicitly discusses the limitations of approximate Shapley attribution and reports its computational overhead.
Critical weaknesses
  1. Both instance-level and abstract procedural memories have been explored in prior work, including D2Skill and XSkill. Similarly, Shapley-based attribution is an established technique. DG-Mem primarily combines these existing ideas, and the paper does not sufficiently establish what new insight or capability emerges from their integration beyond improved empirical performance.
  2. Although utility weighting improves performance in Table 3, its gains are relatively small compared with those from memory construction. More importantly, the paper does not compare Shapley-based ranking against simpler alternatives, such as individual rule accuracy gains, leave-one-out utility, or frequency-based ranking. Therefore, it remains unclear whether modeling interactions through Shapley values is necessary.
  3. Shapley utilities are estimated using the same training partition from which the schema rules are synthesized, raising concerns about their transferability to unseen problems. Moreover, the limited coalition budget and zero-imputation of unevaluated subsets may introduce biased attribution estimates, while requiring approximately 7–8 additional model evaluations per training problem.
Questions for authors
  1. Is Shapley attribution genuinely more effective than simpler utility estimation strategies? Could the authors compare it with leave-one-out attribution, standalone rule effectiveness, and random or frequency-based ranking under the same offline computation budget?
  2. How well do the estimated rule utilities generalize beyond the data used for memory construction? Would the improvements persist if schema synthesis and Shapley utility estimation were performed on disjoint training subsets, and how sensitive are the results to the coalition evaluation budget?
Reviewer_uNitOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission focuses on the external memory for LLMs. The main point is that the memory should balance the specific visual and logical grounding of individual cases, which is important for generalized, transferable procedures. To futher enhance memory usage, the submission uses both instance-grounded exemplars and category-level schemas directly from a frozen model’s training rollouts. Also, a secondary offline pass evaluates subsets of these rules to assign Shapley-based utilities. The method keeps the model parameters fixed during both memory construction and inference. To validate the effectiveness of the proposed method, the work reports consistent improvements on MathVista, MMMU, MMMU-Pro, and DG-Mem. The experiments also include comparisons with existing memory methods and ablation studies to examine the contribution of each component.

Critical strengths
  • Good motivation and practical framework. The combination of instance-level exemplars and category-level schemas is well motivated. The former preserves useful reasoning strategies and verification steps from individual problems, while the latter summarizes more general guidance across similar problems. I think this is reasonable and makes the memory well-structured for later use.

  • The experimental results are good. First, the ablation results show that combining both types of memory generally performs better than using either alone. This is helpful for understanding the dual-grained agent memory. Second, the submission evaluates the method on MathVista, MMMU, and MMMU-Pro across four different model backbones. The results show consistent improvements in most settings. Also, the submission includes comparisons with existing memory methods, component ablations, and repeated evaluations across different data splits.

  • Interesting use of Shapley attribution. Unlike the common way of similarity-based memory retrieval, this submission studies how individual rules contribute to answer correctness. Evaluating different rule combinations also considers interactions between retrieved memories

Critical weaknesses
  • Please clarify the contribution. The main components, including dual-grained memory and reflection-based memory construction, are closely related to existing methods. I think using dual-grained agent memory is straightforward, but the main contribution seems somewhat weak to me.

  • Insufficient justification for Shapley attribution. Although the experiments show improvements from Shapley-based utility weighting, the gains are relatively small compared with those from memory construction. How about the comparison with simple baselines, such as average correctness gains?

  • Potential high computational cost. Memory construction requires multiple reasoning rollouts, reflection steps, and additional evaluations of rule combinations. However, the paper does not provide a sufficiently detailed comparison of computational costs against the performance gains. It would be better to include such a discussion

Questions for authors
  • Limited analysis of the quality of the generated rules. Since these rules are automatically extracted and summarized by LLMs, they may contain incorrect, redundant, or overly general guidance.

  • How well does the estimated memory utility generalize across problems? Since each rule receives a fixed utility score based on training problems, could the authors analyze whether its contribution remains consistent across different problem categories? I think problem-dependent utility estimation would be more effective.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.