Dual-Grained Agent Memory and Shapley Context Attribution for Multimodal Agentic Learner

External memory allows frozen multimodal models to bypass costly parameter up- dates by leveraging labeled training data at inference. Designing effective memory, however, requires balancing the specific visual and logical grounding of individual cases with the need for generalized, transferable procedures. Furthermore, jointly retrieving multiple memories complicates selection by frequently introducing redun- dant or contradictory context. To address these challenges, we introduce DG-Mem, an offline memory framework that extracts dual-grained representations—instance- grounded exemplars and category-level schemas—directly from a frozen model’s training rollouts. Per-problem reflections are dynamically clustered by an incre- mental categorizer and synthesized into shared procedural advice. To optimize test-time ranking and resolve retrieval conflicts, a secondary offline pass evaluates subsets of these rules to assign Shapley-based utilities. Evaluated on held-out parti- tions of MathVista, MMMU, and MMMU-Pro, DG-Mem consistently improves accuracy over retrieval-free prompting across both open-weight and proprietary backbones. The framework maintains this advantage across repeated trials and outperforms tool-free XSkill under randomized splits. Applying DG-Mem to text- only mathematics yields similar gains: memory constructed from DAPO problems improves pooled AIME and HMMT accuracy, surpassing TFGRPO. Ablations confirm that the dual memory grains function complementarily, and Shapley utility weighting successfully provides further incremental gains. Ultimately, DG-Mem validates offline memory synthesis as a highly effective, practical adaptation strat- egy for frozen models, establishing a strong foundation for future advancements in rule-level context attribution.