- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents CaliMem, a memory-based agentic framework for adaptive data curation. The authors first demonstrate that fixed data curation strategies can have substantially different effects across downstream models and tasks, motivating the use of iterative agentic search to identify task-specific curation policies.
CaliMem extends a basic propose–evaluate–revise agent with persistent memory and prospective performance prediction. Before evaluating a proposed policy, the agent predicts its downstream performance based on accumulated evidence and claims. The discrepancy between the predicted and observed outcomes is then used to revise the claims stored in memory, with the goal of avoiding overgeneralization and improving subsequent exploration.
The method is evaluated across 55 biological dataset–model tasks and several LLM post-training settings. The results show improvements over fixed data curation methods and a basic agentic baseline, with additional experiments investigating search efficiency, prediction quality, and the contributions of memory and prediction.
Critical strengths
-
The paper identifies a meaningful limitation of fixed data curation strategies and motivates adaptive, task-specific search. The evaluation spans several biological domains and LLM post-training tasks, providing evidence that data selection policies can behave differently across downstream settings. This makes the problem relevant beyond a single application.
-
The experimental results are extensive and generally positive. CaliMem is evaluated across multiple tasks, datasets, and agent backbones, and consistently improves over the Basic Agent in most settings. The search trajectories and ablation experiments also provide useful evidence that combining persistent memory with prospective prediction can improve search efficiency under a fixed iteration budget.
Critical weaknesses
-
The proposed calibration mechanism is not sufficiently specified or validated. CaliMem computes a simple residual between the predicted and observed scores, but provides no explicit criterion for assessing prediction reliability or attributing prediction errors to specific memory claims. Instead, the interpretation of prediction errors and the revision of claims are largely left to the LLM. While the ablation results demonstrate the benefits of combining prediction with persistent memory, they do not establish whether these improvements arise from the proposed calibration mechanism or from additional structured reflection on experimental feedback.
-
The experimental comparisons with existing agentic approaches are insufficient. While the paper provides extensive comparisons with conventional data curation methods and evaluates several agent backbones, its evaluation of agentic search primarily relies on the authors' own Basic Agent and its ablated variants. Several relevant approaches, such as AIDE [1] and DataMaster [2], are already cited in the paper but are not included as experimental baselines. These methods also explore iterative experimentation and alternative search strategies. Without comparisons against such approaches under comparable experimental budgets, it is difficult to assess whether CaliMem provides meaningful improvements over existing agentic search methods.
[1] Jiang, Z., et al. (2025). AIDE: AI-Driven Exploration in the Space of Code. arXiv:2502.13138.
[2] Du, Y., et al. (2026). DataMaster: Data-Centric Autonomous AI Research. arXiv:2605.10906.
Questions for authors
-
How does CaliMem distinguish between prediction errors caused by incorrect memory claims and those arising from interactions among policy components or stochastic variation in downstream training? Are there specific guidelines or mechanisms used to determine which claims should be revised?
-
To what extent are the improvements attributable to making explicit numerical predictions, rather than to structured reflection over experimental feedback? Have the authors considered a memory-based baseline that revises claims after each experiment without requiring prospective score predictions?
-
What are the main practical differences between CaliMem and existing agentic experimentation frameworks such as AIDE and DataMaster? Were there specific reasons for not including adapted versions of these approaches as experimental baselines under the same search budget?
5 comments
W1 / Q2: Unclear Whether Prediction Improves Performance Beyond Structured Reflection
We thank the reviewer for raising this question. The requested memory-based baseline is already included in our ablation study (Figure 6). We clarify below how this comparison evaluates the additional contribution of prediction-based calibration.
1. Memory-only vs. CaliMem. As described in Section 3.3, Memory-only maintains persistent evidence and claims, and revises its claims after each experiment based on the observed feedback, without making prospective score predictions. CaliMem follows the same memory update process but additionally predicts policy performance before evaluation and uses the difference between predicted and observed scores to guide claim revision. Therefore, Memory-only already provides the structured reflection baseline requested by the reviewer.
2. Experimental results. As shown in Figure 6, on protein variant-effect prediction, Memory-only achieves 39.05, while CaliMem achieves 39.59 (+0.54). On batch integration, Memory-only achieves 71.24, while CaliMem achieves 72.89 (+1.65). Since Memory-only already revises claims based on experimental feedback, these additional gains support the benefit of prediction-based calibration beyond structured reflection alone.
W2 / Q3: Insufficient Comparison with Existing Agentic Search Methods
We thank the reviewer for raising this concern. We first clarify the methodological differences between CaliMem and AIDE/DataMaster, and then add both methods as baselines under the same search budget. The results show that CaliMem achieves higher performance than both methods across all five biology tasks.
1. Methodological differences. AIDE uses tree-based search to iteratively improve ML solutions through code optimization, while DataMaster combines external data discovery and data curation with tree-structured search and global memory. As discussed in Section 2, CaliMem addresses a complementary question: what does an agent learn about downstream performance over a long search trajectory, and when should it trust what it has learned? Unlike AIDE and DataMaster, CaliMem explicitly uses prospective predictions and observed outcomes to reassess the claims stored in memory, helping the agent refine its understanding of how data-curation policies affect downstream performance.
2. Additional baseline comparisons. We add AIDE and DataMaster as baselines across all five biology tasks. For AIDE, we use its tree-based search to optimize data-curation policies instead of ML code. Since our study focuses on data curation within a fixed data pool, we disable DataMaster's external data discovery (Red Nodes) and retain its data-curation operations (Black Nodes), tree-structured search, and Global Memory. All methods use Claude Opus 4.8 as the underlying model, with the same data pool, action space, downstream training pipeline, and search budget of 15 iterations. We report held-out test performance as mean ± standard deviation across three independent runs. The results are shown in the table below.
Table 1. Held-Out Test Performance Across Five Biology Tasks (Mean ± STD).
| Method | Protein | Non-coding | Cell type | Batch | Perturbation |
|---|---|---|---|---|---|
| Full data | 44.72 | 13.47 | 62.01 | 68.55 | 46.29 |
| Random 50% | 42.25 ± 0.48 | 12.49 ± 0.26 | 61.52 ± 0.37 | 68.27 ± 0.14 | 45.54 ± 0.32 |
| Basic Agent | 45.86 ± 0.53 | 13.79 ± 0.21 | 63.86 ± 0.58 | 68.92 ± 0.17 | 47.13 ± 0.46 |
| AIDE | 45.72 ± 0.49 | 13.89 ± 0.23 | 63.74 ± 0.54 | 69.04 ± 0.11 | 46.99 ± 0.34 |
| DataMaster | 46.19 ± 0.41 | 14.30 ± 0.62 | 64.72 ± 0.23 | 69.97 ± 0.31 | 47.73 ± 0.22 |
| CaliMem | 46.58 ± 0.32 | 14.77 ± 0.20 | 65.35 ± 0.38 | 70.84 ± 0.18 | 49.09 ± 0.33 |
As shown in the table, CaliMem achieves the highest mean test performance across all five biology tasks. AIDE's tree-based search is designed for code optimization, where it can improve solutions by modifying code and comparing execution results. However, in data curation, a strategy that performs poorly in one experiment may work well when combined with other policy components. Relying mainly on observed scores may therefore make it difficult to identify promising strategies for further exploration. DataMaster maintains global memory of past experiments, but does not explicitly use prediction errors to check and revise its previous conclusions. CaliMem addresses this limitation by comparing predicted and observed performance to reassess the claims stored in memory, helping the agent make better decisions in subsequent iterations.
W1 / Q1: Prediction Errors and Memory Claim Revision
We thank the reviewer for raising this question. CaliMem uses prediction errors as signals to reassess memory claims based on experimental evidence, rather than to identify the exact cause of each prediction error. The goal is to prevent overgeneralized claims from misleading subsequent data curation search.
Purpose of calibration. As described in Section 3.4, before evaluating a policy, the agent records its predicted score, the evidence and claims supporting that prediction, and an outcome that would challenge those claims. After evaluation, the agent compares the predicted and observed scores to check whether the policy performed as expected. The discrepancy prompts the agent to reassess the claims used to make the prediction.
How claims are revised. As explicitly stated in Section 3.4, "a discrepancy prompts reassessment but does not by itself identify the cause." A prediction may rely on multiple claims, while the evaluation returns only one score. This score alone cannot identify which claim is incorrect, especially when different policy components may interact. CaliMem therefore uses the LLM to examine the executed policy, previous experimental results, and the claims behind the prediction before deciding which claims, if any, need to be revised. Figure 2 illustrates this process. When rebalancing reduces performance in combination with a confidence filter, CaliMem revises its claim to reflect that specific configuration instead of concluding that rebalancing is generally ineffective. This allows the agent to continue exploring rebalancing under other conditions.
Downstream training variation. As described in the Reproducibility Statement, we use fixed random seeds and fixed training and evaluation pipelines for downstream models. For code instruction tuning, we also use deterministic kernels. These settings largely eliminate variation caused by randomness in downstream training.
Empirical validation. Section 5.2 evaluates the contribution of prediction-based calibration through ablation experiments. The Memory-only baseline already maintains and revises claims based on experimental feedback, but does not make predictions before evaluation. CaliMem adds this prediction step and uses the observed discrepancy to reassess memory claims. This improves performance from 39.05 to 39.59 on protein variant-effect prediction and from 71.24 to 72.89 on batch integration. Section 5.1 further shows that memory becomes more accurate at predicting the performance of future policies as more experimental results are accumulated. These results support the effectiveness of prediction-based calibration in improving memory and guiding subsequent data curation search.