acceptodds
Under review as a conference paper at ICLR 2027

CaliMem: Search-Efficient Data Agent via Calibrated Memory

Abstract

Data curation is a first-order design choice in machine learning, yet tailoring training data to a downstream model and task remains costly in labor, expertise, and compute. Datasets are therefore often processed through fixed pipelines and reused across applications. We show that this assumption is fragile: across 55 dataset–model tasks spanning biological domains, the same curation recipe can help one model while hurting another. Effective curation must adapt to the downstream setting, motivating automated search over data recipes. We first build an agentic system that searches for curation policies at deployment, improving data utility via test-time curation (TTC). Although achieving performance gains, its search is driven largely by recent feedback and remains local. Adding persistent memory helps the agent retain evidence across iterations and reveals a deeper problem: it forms claims that extend beyond the conditions in which they were observed, sometimes abandoning an entire strategy after a single unsuccessful trial. To address this problem, we introduce CaliMem, a calibrated-memory agentic loop that requires the agent to predict the outcome of each proposed policy before executing it. Comparing this prediction with the observed result gives the agent a reality check on the claims behind its proposal. It uses the prediction error to retain supported claims, revise mistaken ones, and specify the conditions under which a conclusion holds. This makes the agent's beliefs testable, reduces overgeneralization, and supports more effective long-horizon exploration. Across biological and LLM post-training tasks, CaliMem consistently outperforms fixed curation baselines and published methods, and exceeds full-data training by up to 19.4%. Code is open-sourced.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Agentic data curation, LLM agents, Biomedical machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QYsUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents CaliMem, a memory-based agentic framework for adaptive data curation. The authors first demonstrate that fixed data curation strategies can have substantially different effects across downstream models and tasks, motivating the use of iterative agentic search to identify task-specific curation policies.

CaliMem extends a basic propose–evaluate–revise agent with persistent memory and prospective performance prediction. Before evaluating a proposed policy, the agent predicts its downstream performance based on accumulated evidence and claims. The discrepancy between the predicted and observed outcomes is then used to revise the claims stored in memory, with the goal of avoiding overgeneralization and improving subsequent exploration.

The method is evaluated across 55 biological dataset–model tasks and several LLM post-training settings. The results show improvements over fixed data curation methods and a basic agentic baseline, with additional experiments investigating search efficiency, prediction quality, and the contributions of memory and prediction.

Critical strengths
  1. The paper identifies a meaningful limitation of fixed data curation strategies and motivates adaptive, task-specific search. The evaluation spans several biological domains and LLM post-training tasks, providing evidence that data selection policies can behave differently across downstream settings. This makes the problem relevant beyond a single application.

  2. The experimental results are extensive and generally positive. CaliMem is evaluated across multiple tasks, datasets, and agent backbones, and consistently improves over the Basic Agent in most settings. The search trajectories and ablation experiments also provide useful evidence that combining persistent memory with prospective prediction can improve search efficiency under a fixed iteration budget.

Critical weaknesses
  1. The proposed calibration mechanism is not sufficiently specified or validated. CaliMem computes a simple residual between the predicted and observed scores, but provides no explicit criterion for assessing prediction reliability or attributing prediction errors to specific memory claims. Instead, the interpretation of prediction errors and the revision of claims are largely left to the LLM. While the ablation results demonstrate the benefits of combining prediction with persistent memory, they do not establish whether these improvements arise from the proposed calibration mechanism or from additional structured reflection on experimental feedback.

  2. The experimental comparisons with existing agentic approaches are insufficient. While the paper provides extensive comparisons with conventional data curation methods and evaluates several agent backbones, its evaluation of agentic search primarily relies on the authors' own Basic Agent and its ablated variants. Several relevant approaches, such as AIDE [1] and DataMaster [2], are already cited in the paper but are not included as experimental baselines. These methods also explore iterative experimentation and alternative search strategies. Without comparisons against such approaches under comparable experimental budgets, it is difficult to assess whether CaliMem provides meaningful improvements over existing agentic search methods.

[1] Jiang, Z., et al. (2025). AIDE: AI-Driven Exploration in the Space of Code. arXiv:2502.13138.

[2] Du, Y., et al. (2026). DataMaster: Data-Centric Autonomous AI Research. arXiv:2605.10906.

Questions for authors
  1. How does CaliMem distinguish between prediction errors caused by incorrect memory claims and those arising from interactions among policy components or stochastic variation in downstream training? Are there specific guidelines or mechanisms used to determine which claims should be revised?

  2. To what extent are the improvements attributable to making explicit numerical predictions, rather than to structured reflection over experimental feedback? Have the authors considered a memory-based baseline that revises claims after each experiment without requiring prospective score predictions?

  3. What are the main practical differences between CaliMem and existing agentic experimentation frameworks such as AIDE and DataMaster? Were there specific reasons for not including adapted versions of these approaches as experimental baselines under the same search budget?

5 comments

W1 / Q1: Prediction Errors and Memory Claim Revision

We thank the reviewer for raising this question. CaliMem uses prediction errors as signals to reassess memory claims based on experimental evidence, rather than to identify the exact cause of each prediction error. The goal is to prevent overgeneralized claims from misleading subsequent data curation search.

Purpose of calibration. As described in Section 3.4, before evaluating a policy, the agent records its predicted score, the evidence and claims supporting that prediction, and an outcome that would challenge those claims. After evaluation, the agent compares the predicted and observed scores to check whether the policy performed as expected. The discrepancy prompts the agent to reassess the claims used to make the prediction.

How claims are revised. As explicitly stated in Section 3.4, "a discrepancy prompts reassessment but does not by itself identify the cause." A prediction may rely on multiple claims, while the evaluation returns only one score. This score alone cannot identify which claim is incorrect, especially when different policy components may interact. CaliMem therefore uses the LLM to examine the executed policy, previous experimental results, and the claims behind the prediction before deciding which claims, if any, need to be revised. Figure 2 illustrates this process. When rebalancing reduces performance in combination with a confidence filter, CaliMem revises its claim to reflect that specific configuration instead of concluding that rebalancing is generally ineffective. This allows the agent to continue exploring rebalancing under other conditions.

Downstream training variation. As described in the Reproducibility Statement, we use fixed random seeds and fixed training and evaluation pipelines for downstream models. For code instruction tuning, we also use deterministic kernels. These settings largely eliminate variation caused by randomness in downstream training.

Empirical validation. Section 5.2 evaluates the contribution of prediction-based calibration through ablation experiments. The Memory-only baseline already maintains and revises claims based on experimental feedback, but does not make predictions before evaluation. CaliMem adds this prediction step and uses the observed discrepancy to reassess memory claims. This improves performance from 39.05 to 39.59 on protein variant-effect prediction and from 71.24 to 72.89 on batch integration. Section 5.1 further shows that memory becomes more accurate at predicting the performance of future policies as more experimental results are accumulated. These results support the effectiveness of prediction-based calibration in improving memory and guiding subsequent data curation search.

W1 / Q2: Unclear Whether Prediction Improves Performance Beyond Structured Reflection

We thank the reviewer for raising this question. The requested memory-based baseline is already included in our ablation study (Figure 6). We clarify below how this comparison evaluates the additional contribution of prediction-based calibration.

1. Memory-only vs. CaliMem. As described in Section 3.3, Memory-only maintains persistent evidence and claims, and revises its claims after each experiment based on the observed feedback, without making prospective score predictions. CaliMem follows the same memory update process but additionally predicts policy performance before evaluation and uses the difference between predicted and observed scores to guide claim revision. Therefore, Memory-only already provides the structured reflection baseline requested by the reviewer.

2. Experimental results. As shown in Figure 6, on protein variant-effect prediction, Memory-only achieves 39.05, while CaliMem achieves 39.59 (+0.54). On batch integration, Memory-only achieves 71.24, while CaliMem achieves 72.89 (+1.65). Since Memory-only already revises claims based on experimental feedback, these additional gains support the benefit of prediction-based calibration beyond structured reflection alone.

W2 / Q3: Insufficient Comparison with Existing Agentic Search Methods

We thank the reviewer for raising this concern. We first clarify the methodological differences between CaliMem and AIDE/DataMaster, and then add both methods as baselines under the same search budget. The results show that CaliMem achieves higher performance than both methods across all five biology tasks.

1. Methodological differences. AIDE uses tree-based search to iteratively improve ML solutions through code optimization, while DataMaster combines external data discovery and data curation with tree-structured search and global memory. As discussed in Section 2, CaliMem addresses a complementary question: what does an agent learn about downstream performance over a long search trajectory, and when should it trust what it has learned? Unlike AIDE and DataMaster, CaliMem explicitly uses prospective predictions and observed outcomes to reassess the claims stored in memory, helping the agent refine its understanding of how data-curation policies affect downstream performance.

2. Additional baseline comparisons. We add AIDE and DataMaster as baselines across all five biology tasks. For AIDE, we use its tree-based search to optimize data-curation policies instead of ML code. Since our study focuses on data curation within a fixed data pool, we disable DataMaster's external data discovery (Red Nodes) and retain its data-curation operations (Black Nodes), tree-structured search, and Global Memory. All methods use Claude Opus 4.8 as the underlying model, with the same data pool, action space, downstream training pipeline, and search budget of 15 iterations. We report held-out test performance as mean ± standard deviation across three independent runs. The results are shown in the table below.

Table 1. Held-Out Test Performance Across Five Biology Tasks (Mean ± STD).

MethodProteinNon-codingCell typeBatchPerturbation
Full data44.7213.4762.0168.5546.29
Random 50%42.25 ± 0.4812.49 ± 0.2661.52 ± 0.3768.27 ± 0.1445.54 ± 0.32
Basic Agent45.86 ± 0.5313.79 ± 0.2163.86 ± 0.5868.92 ± 0.1747.13 ± 0.46
AIDE45.72 ± 0.4913.89 ± 0.2363.74 ± 0.5469.04 ± 0.1146.99 ± 0.34
DataMaster46.19 ± 0.4114.30 ± 0.6264.72 ± 0.2369.97 ± 0.3147.73 ± 0.22
CaliMem46.58 ± 0.3214.77 ± 0.2065.35 ± 0.3870.84 ± 0.1849.09 ± 0.33

As shown in the table, CaliMem achieves the highest mean test performance across all five biology tasks. AIDE's tree-based search is designed for code optimization, where it can improve solutions by modifying code and comparing execution results. However, in data curation, a strategy that performs poorly in one experiment may work well when combined with other policy components. Relying mainly on observed scores may therefore make it difficult to identify promising strategies for further exploration. DataMaster maintains global memory of past experiments, but does not explicitly use prediction errors to check and revise its previous conclusions. CaliMem addresses this limitation by comparing predicted and observed performance to reassess the claims stored in memory, helping the agent make better decisions in subsequent iterations.

Reviewer_hujDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper frames data curation as an iterative search problem within a fixed downstream training and evaluation pipeline. It proposes CALIMEM, in which an agent predicts the score of a candidate curation policy before running it, then uses the prediction error to update the evidence and conditional claims stored in memory. On 55 biological dataset–model pairs and three LLM post-training settings, the authors compare CALIMEM against fixed curation policies and agents without calibrated memory, and report higher best scores and faster search progress.

Critical strengths
  1. The paper optimizes data curation directly for a specific downstream model and task instead of assuming a universal curation recipe, which makes for a practically meaningful problem setting. The experiments span several biology tasks, LLM post-training, and multiple agent backends, which suggests fairly broad applicability.

  2. The appendix includes concrete trajectories, memory contents, curation policies, and training configurations. This makes the agents' behavior reasonably auditable and is more informative than reporting final scores alone.

Critical weaknesses
  1. The evaluation protocol is unclear, and the results may suffer from adaptive selection bias. Section 4 says the agent observes scores on a "held-out test split", but Section 3.1 and Appendix B.1 indicate that metrics are computed on an "evaluation split", and the appendix defines a separate "test split" for some tasks. The headline numbers are the best scores over 10–15 iterations, yet the final selected policy is not evaluated on data withheld from the agent's feedback. Under either reading, the results do not support the generalization claims. If the agent sees test scores, it is searching adaptively on the test set and reporting the maximum. If it sees evaluation scores, there is no truly held-out result. This affects central claims such as outperforming full-data training and fixed baselines. It is especially concerning for the LLM experiments, where the average over several benchmarks is used repeatedly to guide the search and no independent evaluation set is provided.

  2. The gains cannot be cleanly attributed to predictive calibrated memory, and uncertainty is not quantified. In cell-type annotation and batch integration, the Basic agent and CALIMEM operate in different action spaces: CALIMEM can also modify the gene mask, so part of its advantage may come from the larger action space. In addition, the main results appear to come from a single search trajectory per setting, with no reported number of independent runs, variance, or confidence intervals. The agent is stochastic, and fixing the training seed does not remove that variability. The forecasting experiment in Section 5.1 also falls short of establishing the role of calibration. It does not compare against alternative representations of the same information (e.g., the raw history, Basic-agent logs, or a simple summary), and it predicts later policies from the same adaptive trajectory, which the agent had already chosen.

Questions for authors
  1. Which split provides the score the agent observes at each iteration for every task type? Are the "held-out test split" in Section 4 and the "evaluation split" in the appendix the same thing? If not, please report results where the search uses evaluation feedback only and each method's final frozen policy is evaluated once on a test split that never provided feedback.

  2. For cell-type annotation and batch integration, please run the Basic agent with the same action space as CALIMEM, and provide an action-space-matched comparison of Basic, memory-only, prediction-only, and CALIMEM.

  3. How many independent runs were conducted for each agent–task–method combination? Please report the variance across runs, along with confidence intervals or significance tests for CALIMEM relative to Basic.

6 comments

W1 / Q1. Inconsistent evaluation terminology and lack of held-out test results.

1. For the five biology tasks, each dataset has training, validation, and test splits. The "held-out test split" in Section 4 (Experiment Setup) and the "evaluation split" in Section 3.1 and Appendix B.1 both refer to the validation split. In the revised paper, we consistently refer to this as the "search split", which provides the feedback score observed by the agent at each iteration.

2. To assess generalization beyond the search split, we conducted additional evaluations on all five biology tasks. For each independent run, we select the policy with the highest search-split score and evaluate it once on the test split, which is never used for agent feedback or policy selection. All baselines are evaluated on the same test split. For the LLM post-training tasks, we partition each benchmark into search and test splits at a 1:2 ratio using a fixed random seed, and follow the same procedure with a 10-iteration search. We repeat each agent–task configuration three times with independent search runs and report the mean ± standard deviation of the test scores. The results are shown below and The best result among agent variants for each task is highlighted in bold.

Table 1. Held-Out Test Performance Across Five Biology Tasks.

ModelAgent variantProteinNon-codingCell typeBatchPerturbation
Full data (baseline)–44.7213.4762.0168.5546.29
Random 50%–42.25 ± 0.4812.49 ± 0.2661.52 ± 0.3768.27 ± 0.1445.54 ± 0.32
Fixed pipeline 50%, avg. of 12 methods–41.85 ± 1.3612.55 ± 0.4161.43 ± 0.8368.20 ± 0.3145.39 ± 1.12
Fixed pipeline 50%, best per subtask–45.37 ± 0.3913.42 ± 0.1862.64 ± 0.4468.71 ± 0.1346.66 ± 0.27
Claude Code – select 50%
Opus 4.8Basic agent45.86 ± 0.5313.79 ± 0.2163.86 ± 0.5868.92 ± 0.1747.13 ± 0.46
CaliMem46.58 ± 0.3214.77 ± 0.2065.35 ± 0.3870.84 ± 0.1849.09 ± 0.33
Sonnet 5Basic agent45.56 ± 0.4113.76 ± 0.3263.48 ± 0.4769.04 ± 0.1247.02 ± 0.38
CaliMem46.89 ± 0.3314.48 ± 0.2265.27 ± 0.3570.72 ± 0.1748.83 ± 0.31
Haiku 4.5Basic agent44.97 ± 0.7213.54 ± 0.2462.82 ± 0.6168.88 ± 0.2846.81 ± 0.57
CaliMem45.25 ± 0.3813.98 ± 0.1864.42 ± 0.3670.38 ± 0.2047.91 ± 0.32
Codex – select 50%
GPT-5.3-CodexBasic agent44.68 ± 0.5613.63 ± 0.2763.49 ± 0.4368.79 ± 0.2246.98 ± 0.51
CaliMem45.47 ± 0.3614.12 ± 0.1863.91 ± 0.3470.55 ± 0.1748.72 ± 0.35
GPT-5.6 SolBasic agent45.11 ± 0.4713.73 ± 0.2363.19 ± 0.5268.94 ± 0.1847.16 ± 0.42
CaliMem46.55 ± 0.3014.62 ± 0.2564.46 ± 0.3271.39 ± 0.2348.96 ± 0.27
OpenHands – select 50%
Qwen3.5-397B-A17BBasic agent45.35 ± 0.6513.81 ± 0.3462.76 ± 0.6869.07 ± 0.2546.92 ± 0.49
CaliMem45.82 ± 0.3314.37 ± 0.2364.85 ± 0.3670.47 ± 0.2248.28 ± 0.31

Table 2. Held-Out Test Performance Across Three LLM Post-Training Tasks.

ModelAgent variantOLMo-2-1B (1.1%)Llama-2-7B (3.7%)Qwen2.5-Coder-1.5B (0.2%)
Base model (no fine-tuning)–29.7042.8157.87
Random (baseline)–30.16 ± 0.3843.27 ± 0.3158.19 ± 0.22
Entire dataset–30.0445.1261.08
LESS–31.8844.3058.12
DELIFT–31.1343.3458.41
Claude Code
Opus 4.8Basic agent30.82 ± 0.3444.03 ± 0.3958.62 ± 0.25
CaliMem31.76 ± 0.2344.98 ± 0.3059.54 ± 0.18
Sonnet 5Basic agent30.47 ± 0.4243.88 ± 0.2758.34 ± 0.32
CaliMem31.29 ± 0.2644.63 ± 0.2959.28 ± 0.19
Haiku 4.5Basic agent30.58 ± 0.5143.79 ± 0.4358.43 ± 0.28
CaliMem30.11 ± 0.3143.76 ± 0.2758.82 ± 0.23
Codex
GPT-5.6 SolBasic agent30.94 ± 0.2644.12 ± 0.3558.78 ± 0.23
CaliMem31.63 ± 0.2845.24 ± 0.2359.82 ± 0.19
GPT-5.3-CodexBasic agent30.35 ± 0.4443.91 ± 0.3258.29 ± 0.30
CaliMem30.96 ± 0.2444.74 ± 0.2758.91 ± 0.20
OpenHands
Qwen3.5-397B-A17BBasic agent29.91 ± 0.4743.69 ± 0.3858.16 ± 0.35
CaliMem30.39 ± 0.2544.48 ± 0.2058.86 ± 0.21

On the held-out test splits, CaliMem outperforms the Basic agent in 46 out of 48 comparisons, including 30/30 biology cases and 16/18 LLM post-training cases. For biology tasks, CaliMem also surpasses full-data training in all 30 cases while using only 50% of the training data. For LLM post-training, CaliMem outperforms Random in 17/18 cases, selecting only 0.2–3.7% of the training data.

W2 / Q2. Unmatched action spaces and insufficient ablation studies to isolate the contributions of predictive calibrated memory.

We thank the reviewer for pointing out the unmatched action spaces between CaliMem and the Basic agent. We reran Basic with the same action space as CaliMem on held-out test sets and found that Basic performs slightly worse than with its original action space. This suggests that CaliMem's improvements come from predictive calibrated memory rather than the expanded action space. Further analysis of the search traces shows that the core issue is Basic's tendency to generalize individual failures into conclusions about entire search directions, which limits further exploration.

1. We reran basic agent with the same action space as CaliMem, allowing both to modify cell selection and the gene mask. We also compare Prediction-only and Memory-only under the same setting. Following W1/Q1, we report test scores across three independent runs (mean ± std).

Cell-type annotation (macro-F1)

Agent variantAction spaceTest
Basic (original)cells63.86 ± 0.58
Basiccells + gene mask63.61 ± 0.55
Prediction-onlycells + gene mask64.12 ± 0.49
Memory-onlycells + gene mask64.82 ± 0.44
CaliMemcells + gene mask65.35 ± 0.3

Batch integration (scIB overall)

Agent variantAction spaceTest
Basic (original)cells68.92 ± 0.17
Basiccells + gene mask68.79 ± 0.22
Prediction-onlycells + gene mask69.17 ± 0.20
Memory-onlycells + gene mask69.86 ± 0.19
CaliMemcells + gene mask70.84 ± 0.18

Across both tasks, CaliMem achieves the highest test scores, reaching 65.35 ± 0.38 on cell-type annotation and 70.84 ± 0.18 on batch integration, outperforming the action-space-matched Basic agent by 1.74 and 2.05 points, respectively. Expanding the action space alone does not improve Basic's performance (63.86 → 63.61 and 68.92 → 68.79). These results indicate that CaliMem's improvement comes from predictive calibrated memory rather than a larger action space.

2. To further examine this behavior, we compare two runs on the same batch integration task (lung, Harmony), using identical action spaces and 15-iteration budgets. The full search traces are provided in Appendix C.

Basic: After two unsuccessful cell-selection attempts at iterations 2–3, Basic concluded that "random cells optimal" and stopped exploring cell selection. It then spent seven iterations modifying gene panels, with a maximum improvement of only 0.001, below the scorer's repeatability band of 0.006. The remaining five iterations repeated the same policy without further exploration.

CaliMem: CaliMem also encountered unsuccessful attempts in both cell selection and gene-mask modification. Its predicted scores of 0.690 and 0.700 were followed by observed scores of 0.679 and 0.655. However, these failures did not lead it to abandon either search direction. It continued exploring alternative cell-selection strategies and reached 0.690 at iteration 4 by adjusting cell composition across sequencing protocols.

These results support our motivation in Section 1. Basic tends to generalize individual failures into conclusions about entire search directions. CaliMem uses accumulated feedback and prediction errors to revise such conclusions, enabling more effective exploration. This further shows that CaliMem's improvement comes from predictive calibrated memory rather than a larger action space.

W2 / Q3. Lack of Statistical Analysis

We thank the reviewer for raising this question. To assess CaliMem's stability, we report the average standard deviation across five biology tasks and three LLM post-training tasks based on the held-out test results in W1/Q1 (Tables 3 and 4). We also report 95% confidence intervals and two-sided Welch's t-tests for each agent–task pair in Table 5.

Table 3. Average Standard Deviation of Held-Out Test Performance Across Five Biology Tasks and Three Independent Runs for Each Agent Backbone.

AgentBasic agentCaliMem
Opus 4.80.390.28
Sonnet 50.340.28
Haiku 4.50.480.29
GPT-5.3-Codex0.400.28
GPT-5.6 Sol0.360.27
Qwen3.5-397B-A17B0.480.29
Average0.410.28

Table 4. Average Standard Deviation of Held-Out Test Performance Across Three LLM Post-Training Tasks and Three Independent Runs for Each Agent Backbone.

AgentBasic agentCaliMem
Opus 4.80.330.24
Sonnet 50.340.25
Haiku 4.50.410.27
GPT-5.3-Codex0.350.24
GPT-5.6 Sol0.280.23
Qwen3.5-397B-A17B0.400.22
Average0.350.24

Table 5. Held-Out Test Performance Differences Between CaliMem and Basic Across Agent Backbones.

Mean Difference (Δ) = CaliMem − Basic. We report 95% confidence intervals (CIs) and p-values from two-sided Welch's t-tests. Bold p-values indicate statistical significance (p < 0.05).

(a) Biology tasks

TaskAgent backboneMean Difference (Δ)95% CIp-value
ProteinOpus 4.8+0.72[-0.36, 1.80]0.129
Sonnet 5+1.33[0.47, 2.19]0.013
Haiku 4.5+0.28[-1.21, 1.77]0.593
GPT-5.3-Codex+0.79[-0.35, 1.93]0.121
GPT-5.6 Sol+1.44[0.48, 2.40]0.016
Qwen3.5-397B+0.47[-0.88, 1.82]0.346
Non-codingOpus 4.8+0.98[0.51, 1.45]0.004
Sonnet 5+0.72[0.06, 1.38]0.039
Haiku 4.5+0.44[-0.06, 0.94]0.069
GPT-5.3-Codex+0.49[-0.06, 1.04]0.068
GPT-5.6 Sol+0.89[0.34, 1.44]0.011
Qwen3.5-397B+0.56[-0.14, 1.26]0.086
Cell typeOpus 4.8+1.49[0.30, 2.68]0.027
Sonnet 5+1.79[0.82, 2.76]0.008
Haiku 4.5+1.60[0.35, 2.85]0.026
GPT-5.3-Codex+0.42[-0.48, 1.32]0.259
GPT-5.6 Sol+1.27[0.21, 2.33]0.031
Qwen3.5-397B+2.09[0.69, 3.49]0.018
BatchOpus 4.8+1.92[1.52, 2.32]<0.001
Sonnet 5+1.68[1.33, 2.03]<0.001
Haiku 4.5+1.50[0.92, 2.08]0.002
GPT-5.3-Codex+1.76[1.30, 2.22]<0.001
GPT-5.6 Sol+2.45[1.97, 2.93]<0.001
Qwen3.5-397B+1.40[0.86, 1.94]0.002
PerturbationOpus 4.8+1.96[1.01, 2.91]0.005
Sonnet 5+1.81[1.01, 2.61]0.004
Haiku 4.5+1.10[-0.07, 2.27]0.058
GPT-5.3-Codex+1.74[0.70, 2.78]0.011
GPT-5.6 Sol+1.80[0.94, 2.66]0.006
Qwen3.5-397B+1.36[0.36, 2.36]0.021

(b) LLM post-training tasks

TaskAgent backboneMean Difference (Δ)95% CIp-value
OLMo-2-1BOpus 4.8+0.94[0.24, 1.64]0.021
Sonnet 5+0.82[-0.04, 1.68]0.056
Haiku 4.5-0.47[-1.51, 0.57]0.258
GPT-5.3-Codex+0.61[-0.30, 1.52]0.123
GPT-5.6 Sol+0.69[0.08, 1.30]0.036
Qwen3.5-397B+0.48[-0.49, 1.45]0.215
Llama-2-7BOpus 4.8+0.95[0.14, 1.76]0.032
Sonnet 5+0.75[0.11, 1.39]0.031
Haiku 4.5-0.03[-0.91, 0.85]0.924
GPT-5.3-Codex+0.83[0.15, 1.51]0.028
GPT-5.6 Sol+1.12[0.40, 1.84]0.014
Qwen3.5-397B+0.79[0.01, 1.57]0.049
Qwen2.5-Coder-1.5BOpus 4.8+0.92[0.41, 1.43]0.009
Sonnet 5+0.94[0.29, 1.59]0.019
Haiku 4.5+0.39[-0.20, 0.98]0.138
GPT-5.3-Codex+0.62[0.01, 1.23]0.049
GPT-5.6 Sol+1.04[0.55, 1.53]0.004
Qwen3.5-397B+0.70[-0.02, 1.42]0.053

As shown in Table 5, CaliMem outperforms Basic in 46 out of 48 agent–task pairs, with 32 showing statistically significant improvements (p < 0.05). Surprisingly, on the batch integration task, CaliMem significantly outperforms Basic across all six agent backbones. It also shows lower variability across three independent runs, with the average standard deviation decreasing from 0.41 to 0.28 on the five biology tasks and from 0.35 to 0.24 on the three LLM post-training tasks. These results show that CaliMem achieves consistent and stable performance improvements across different tasks and agent backbones.

Reviewer_VZZmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenge of data curation in machine learning, demonstrating that fixed curation pipelines often fail because a recipe beneficial for one model may harm another. The authors propose Test-Time Curation (TTC), where an automated agent searches for optimal data policies at deployment. They identify two key limitations in existing agentic approaches: local search driven by recent feedback and overgeneralized claims in persistent memory. To solve this, they introduce CALIMEM, a calibrated-memory agentic loop. CALIMEM requires the agent to predict the outcome of a proposed policy before execution. By comparing predictions with actual results, the agent performs a "reality check," revising its internal claims to be conditional and evidence-based. This reduces overgeneralization and enables more effective long-horizon exploration.

Critical strengths

The paper introduces a novel architectural component where the agent must predict the outcome of a proposed policy before evaluation and uses the residual (prediction error) to revise its persistent memory claims. Unlike standard agentic workflows that only update state based on absolute performance improvements (keep/revert), CALIMEM decouples "performance change" from "expectation mismatch." This prevents overgeneralization (e.g., discarding a useful heuristic because it failed in one specific context) by tying failures to their specific conditions rather than ruling out entire directions.

Critical weaknesses

Incomplete or Poorly Organized Supplementary Material (Appendices and Tables)

Questions for authors

See Weaknesses

3 comments

We thank the reviewer for raising this concern. The appendix has been reorganized in the revised supplementary material to improve clarity. We would appreciate the reviewer's feedback on whether any issues remain.

The current appendix contains the following sections, tables, and figures:

SectionTitleTables / Figures
AExtended Related Work–
BExperimental Details
B.1Domains, datasets, models, and metrics (B.1.1–B.1.7: one subsection per task, each with datasets and splits, models, and metrics)Table 2
B.2Data processing baselines for biology tasks (B.2.1 common protocol; B.2.2–B.2.5 one subsection per domain)Figure 7, Tables 3–6
B.3Fine-tuning recipes of the instruction-tuning setupsTables 7–9
CMatched Trajectories of the Basic Agent and CaliMem under the Same Action SpaceTables 10–11
DExamples of Agent Trajectory and Memory Files (D.1 Basic agent; D.2 CaliMem; D.3 memory files)Tables 12–13
EAdditional Empirical ResultsTables 14–15, Figure 8
FForecasting held-out iterations from the CaliMem (F.1 extended results; F.2 prompt)Figures 9–10
GDiscovered Curation Strategies (G.1 Biology; G.2 LLM)–

List of appendix tables

TableContentSection
2Benchmark grid: datasets, pool size, budget, models, headline metric, iterationsB.1
3Protein variant-effect curation recipesB.2.2
4Non-coding variant-effect curation recipesB.2.3
5Cell-type annotation curation recipesB.2.4
6Batch integration curation recipesB.2.5
7Fine-tuning settings, OLMo-2-1B on the Tülu 3 SFT mixtureB.3
8Fine-tuning settings, Llama-2-7B on the LESS mixtureB.3
9Fine-tuning settings, Qwen2.5-Coder-1.5B-Instruct on OpenCodeInstructB.3
10Complete trajectory of the Basic agent, batch integration (lung, Harmony), matched action spaceC
11Complete trajectory of CaliMem on the same task and action spaceC
12Complete trajectory of the Basic agent, perturbation response (Norman19, GEARS)D.1
13Complete trajectory of CaliMem, cell-type annotation (PBMC cross-protocol, SVM)D.2
14Main results on the biology domains (full numbers behind Figure 3 of the main text)E
15Main results on LLM post-training tasks (full numbers behind Figure 4 of the main text)E

List of appendix figures

FigureContentSection
7Standard curation recipes vs. training on unprocessed data, per domainB.2
8Per-task comparison of fixed recipes, the Basic agent, and CaliMemE
9Forecasting error of the frozen memory on later iterations, cell-type annotationF.1
10Forecasting error for non-coding variant effect (two agents) and batch integrationF.1

Dear Reviewer VZZm,

As the extended rebuttal/author discussion period approaches its end tomorrow (11/18), we sincerely look forward to your feedback. We greatly appreciate your valuable time and effort spent reviewing our paper and helping us improve it.

In our response, we have carefully addressed your concern about the organization and completeness of the supplementary material. We have reorganized the appendix to improve its clarity and accessibility, and provided a detailed overview of all sections, tables, and figures to make the experimental details and results easier to follow.

We would be very grateful if you could take another look at our response and the revised supplementary material and let us know whether these changes address your concerns.

Please also let us know if there are any remaining issues or suggestions. We sincerely appreciate your thoughtful review and would welcome any further feedback!

Kind Regards,
Authors of Submission35080

Dear Reviewer VZZm,

As the extended author discussion period ends today (November 18), we would like to follow up once more on our response and revised supplementary material.

We have carefully addressed your concerns by reorganizing the appendix and providing a detailed overview of its contents to make the experimental details and results easier to follow. We would appreciate it if you could review these updates and let us know whether they address your concerns.

If any questions or concerns remain, we would be happy to clarify them before the discussion period closes.

Thank you for your time and consideration.

Best regards,
Authors of Submission35080

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.