acceptodds
Withdrawn from ICLR 2027

AIRE: Closed-Loop Agentic Research for Text Embeddings

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Developing effective text embedding models extends far beyond executing a training script. It requires researchers to navigate interdependent decisions across data curation, objective design, sampling strategies, and evaluation protocols. Because key design trade-offs often emerge only after end-to-end training and retrieval evaluation, embedding development is fundamentally iterative. While recent advances demonstrate that full research workflows can be automated, doing so for text embeddings requires orchestrating domain-specific decisions across data pipeline construction, model training, and retrieval benchmarking. In enterprise settings, this automation must additionally handle distributed user roles, shared accelerator clusters, and strict data governance. To address these challenges, we introduce AIRE (Autonomous Investigation and Research for Embeddings), a multi-agent framework that structures these operations into an autonomous, persistent research loop. In AIRE, a central coordinator delegates literature review, data construction, implementation, training, and evaluation to specialized sub-agents, maintaining a shared cross-round state to guide iterative discovery. We also introduce AIRE-Bench, a benchmark for autonomous embedding research with three task environments built from public retrieval datasets. AIRE-Bench evaluates the resulting model and the research process, tracking model quality, improvement rate, exploration coverage, failure resilience, and compute cost. AIRE achieved an AIRE-Bench score of 89.61.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
text embeddings, autonomous research, multi-agent systems, retrieval, automated experimentation, machine learning systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s3B8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AIRE, a multi-agent framework for automating text embedding research through a closed-loop workflow. AIRE employs a coordinator agent and five specialized agents to handle literature review, data preparation, coding, training, and evaluation.

Critical strengths
  1. The paper presents a reasonably complete framework covering multiple stages of embedding research, from hypothesis generation to experimental evaluation.

  2. AIRE-Bench considers not only the resulting embedding quality but also research trajectories, exploration coverage, reliability, and resource consumption.

Critical weaknesses
  1. The proposed framework primarily integrates existing agentic workflow concepts, including role specialization, persistent memory, iterative experimentation, and artifact-based execution. Similar capabilities have been explored in prior work such as AI Scientist, Agent Laboratory, and other ML research agents.

  2. While the integration is practically useful, the paper does not clearly identify a fundamental challenge in embedding research that existing general-purpose research agents cannot address. The proposed solutions also appear largely based on established engineering practices rather than introducing new mechanisms for research planning, hypothesis selection, or experimental optimization. Consequently, the technical contribution seems relatively narrow and application-specific.

  3. The paper lacks direct comparisons with existing research agents or conventional automated optimization approaches under comparable computational budgets. The evaluation primarily compares the embedding models obtained by AIRE against their initial baselines. However, improvements over an untuned base model do not necessarily demonstrate the effectiveness of the proposed agentic research approach. Similar gains might be achievable through conventional AutoML, hyperparameter optimization, or simpler iterative search strategies.

  4. Each benchmark reports only a single research trajectory, making it difficult to assess statistical reliability and consistency. Although the ablation studies provide some evidence for the usefulness of individual components, they do not establish whether the proposed framework offers meaningful advantages over existing alternatives.

Questions for authors
  1. What are the fundamental challenges specific to embedding research that motivate the proposed architecture?

  2. Could the authors clarify which embedding-specific challenges cannot be effectively addressed by existing approaches, and which components of AIRE provide genuinely new solutions to these challenges?

  3. How does AIRE compare with conventional automated optimization and existing research agents under matched computational budgets?

  4. How robust and efficient is the proposed closed-loop research process?

  5. Comparisons of wall-clock time, GPU-hours, and overall computational cost against alternative approaches would help substantiate the practical benefits of AIRE.

Reviewer_bvVXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AIRE automates text embedding research through a coordinator, five specialist agents, persistent experimental memory, and artifact-based execution controls. It also introduces AIRE-Bench to assess retrieval quality and research trajectories.

Critical strengths

The system connects data curation, training, and evaluation, turning feedback into follow-up experiments.

Frozen specifications, artifact lineage, and failure logs make autonomous runs auditable and recoverable.

Gains across three retrieval domains, supporting ablations, and reported internal deployment suggest practical value.

Critical weaknesses

The paper does not show whether AIRE performs better than standard fine-tuning with the same compute resources.

The paper does not clarify whether the final results use a separate test set that never guided earlier experiments.

Only one full run is reported per task, and ablation repetition counts are unclear, so the experiments do not establish whether the gains are consistent.

Table 2 reports only an overall score for each ablation, making it unclear what each component actually improves.

Questions for authors
  1. Which evaluation split supplies feedback for iterative decisions and model promotion?
  2. How does AIRE compare with conventional fine-tuning methods given the same resources?
  3. How many independent launches support each ablation?
Reviewer_g5idOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents AIRE, a multi-agent system that runs iterative research on text embedding models. A coordinator agent assigns literature review, data construction, coding, training and evaluation to five specialist agents. Each round is stored as a set of artifacts, including failed runs, and the coordinator uses this history to decide the next round. Test relevance judgments are held by a separate evaluation server, and agents receive only aggregated metrics.

The paper also introduces AIRE-Bench: three retrieval tasks (FiQA, NFCorpus, QASPER) that all start from gte-multilingual-base, with a composite score that combines final gain, gain over time and reliability. In one run per task, AIRE improves nDCG@10 by 7.9% to 12.3% over the zero-shot starting model, for a mean score of 89.61. Removing cross-round memory, direction guidance, data intervention or role specialization lowers the mean score to between 66 and 75. The paper also reports a recall gain on an internal product-search model.

Critical strengths

The engineering for auditability is careful. Each round's hypothesis, data, code, configuration and result are frozen and stored. Failures are kept as records instead of being discarded, and a stage counts as complete only when its outputs pass explicit checks (Section 4.4, Appendix B.1). Many automated-research systems do not record this, and it makes AIRE's decisions inspectable after the fact.

The paper is honest about what its results show. Figures 5 and 7 keep failed rounds visible instead of drawing monotone progress. Section 5.3 states that one launch per task cannot establish seed stability or superiority over another search method. This makes the claims easy to assess.

Critical weaknesses
  1. Model selection uses the test set. Section 4.4 says that only the evaluation server reads test relevance judgments. The Promotion Gate then admits a new best model after checking the primary metric from that server. Section 4.2 adds that the Evaluation Agent receives slice-level results. I could not find a separate validation split. Across up to 16 rounds, the reported best nDCG@10 is therefore the maximum of many test evaluations, while the baseline is a single evaluation. This biases the reported gains upward, even though agents never see individual labels. Selecting on a validation split and reporting the selected model once on an untouched test split would fix this.

  2. No comparison isolates what the agentic loop adds. All gains are measured against the zero-shot model. Three comparisons are missing: one standard fine-tuning run on each task's public training data, a random search over the same action space with the same GPU budget, and an existing general ML agent. Without them, the question posed in the introduction is answered only relative to doing nothing. The evidence is also from single runs. Each task is launched once, and Table 2 does not say how many runs each ablation uses (its full-system row repeats Table 1). With stochastic LLM agents, the gap between 89.61 and 74.84 may lie within run-to-run variation.

Both problems need new experiments rather than edits: checkpoint selection on a validation split, at least one simple baseline, and repeated launches. If the authors provide these, I would reconsider my rating.

Minor points (these do not drive my rating).

  • AIRE-Bench is presented as a benchmark, but only AIRE is evaluated on it, so its score has no reference point.
  • The weights in Equation 1 (60, 25, 15) and the ln(1.10) unit are not justified.
  • The internal product-search result cannot be verified, and its compute cost is not reported.
  • Section 5.2 says GPU-hours are tracked, but Table 3 reports only wall-clock hours and LLM tokens.
  • Appendix A.2 is written in the future tense ("The study will publish..."), which reads like text left over from a plan.
Questions for authors

Questions that could change my assessment:

  1. On which data does the Promotion Gate select the best checkpoint? Are these evaluation results also used to guide decisions in later rounds? If there is no validation split, what are the scores when the checkpoint is selected on validation and evaluated once on test?
  2. What nDCG@10 does one standard fine-tuning run reach on each task, from the same starting model and the same training data? What does a random search over the action space reach with the same GPU-hours?
  3. How many runs stand behind each cell of Table 2? What is the spread of the full system's score across independent launches? Please also report nDCG@10 for each ablation condition, not only the composite score.

Clarifications:

  1. How is the retrieval view of QASPER built? Its starting nDCG@10 of 0.0815 is very low.
  2. Does the ordering of the ablations in Table 2 change under other reasonable weights in Equation 1?
  3. The reproducibility statement says the supplement "will include" the source code, and I could not find supplementary material on OpenReview. Will code and run artifacts be available during review?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.