acceptodds
Withdrawn from ICLR 2027

IdeaCompass: Making Exploration Measurable in Autonomous AI Research Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autonomous machine learning (ML) research agents conduct research by proposing ideas, implementing them, and running experiments. Yet existing autoresearch systems lack an explicit quantitative account of the idea landscape they have explored: they do not directly measure how far apart past ideas are, in which directions the search has moved, or how broadly the search has explored the idea space. General-purpose text embeddings do not fill this gap. We find that they do not reliably capture the distance and relative direction between ML research ideas, placing a change in training duration closer to a learning-rate change than to another duration change. We introduce IdeaCompass, a quantitative map of ML ideas: an ML-aware embedding whose vector direction encodes what an idea changes and whose norm encodes how substantial the change is. On this map, an agent's research history becomes a measurable trajectory, from which IdeaCompass diagnoses whether stalled search is over-concentrated (collapse) or over-dispersed, and steers subsequent proposals toward diversification or consolidation accordingly. Across eight FML-bench-Lite tasks, IdeaCompass improves both Autoresearch and AdaptiveSearch, which become the top two of nine evaluated agents; Autoresearch's pairwise win rate rises from 0.52 to 0.71. The representation achieves 0.740 AUROC on direction discrimination and 0.72 Spearman correlation between norm and modification scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
autonomous research agents, LLM agents, automated machine learning, automated scientific discovery, text embeddings, exploration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bhQCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IdeaCompass, a framework designed to make exploration measurable and steerable for autonomous ML research agents. IdeaCompass introduces an ML-specific idea embedding model (fine-tuned from gte-modernbert-base) where the vector direction represents the type of ML modification (10 categories, e.g., optimizer, model architecture, loss function) and the vector norm represents modification magnitude (5 ordinal scopes).

Using this embedding space, the agent's history is considered as a trajectory. A controller dettects search stagnation: when validation improvement stalls over a window W, it checks the trajectory spread, and if the spread is low, it diagnoses Collapse and steers the agent towards diversification; if the spread is high, it diagnoses Dispersion and steers towards consolidation.

Critical strengths
  • This paper has an interesting geometric framing of search dynamics. Formulating exploration trajectory in terms of decoupled direction (what changes) and magnitude (how much changes) is an intuitive idea and can be conceptually sound.
  • Distinguishing between stagnation caused by over-concentration (collapse) and stagnation caused by aimless jumping (dispersion) provides a more nuanced state diagnosis than simple no-progress counters.
  • The paper provides episode-level analyses (Table 4) examining how spread dynamics and episode durations shift under collapse and dispersion interventions.
Critical weaknesses

On potential data leakage,

  • In appendix C lines 648-654, "The representation dataset is collected from 18 FML-bench tasks, including the eight FML-bench-Lite tasks used for downstream evaluation; the representation split is therefore run-disjoint rather than task-disjoint.".
  • The embedding encoder and direction prediction heads are trained on idea proposals collected from the exact same 8 benchmark tasks used for downstream evaluation.
  • Lack of OOD evaluations.

On outlier-driver gains

  • A closer inspection to Table 6 in Appendix B suggests that the performance improvement is mainly from one single task DomainBed-CM, and improvements on other 7 tasks are marginal or negative.
  • The ablation experiments are conducted entirely on the DomainBed-CM task and raises the concern of cherry picking analysis.
Questions for authors

Refer to the above #weaknesses

Reviewer_3RZiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that autonomous ML research agents lack an explicit, quantitative view of the idea space they have explored. It proposes IdeaCompass, an ML-aware idea embedding trained on 33.5K agent-generated ideas labeled by an LLM (Claude Sonnet 5). Each idea gets multi-label “direction” categories (10 classes) and an ordinal “magnitude” level. The embedding is unnormalized: its angle encodes what an idea changes and its norm encodes how large the change is. A controller uses this embedding when a host agent’s search stalls. It measures the dispersion of recent ideas, labels the stall as Collapse or Dispersion via quantile thresholds, injects this diagnosis into the prompt, and selects among K=2 candidates by maximizing or minimizing distance to recent history. On representation tests, the embedding reaches 0.740 AUROC for direction and Spearman 0.72 between norm and scope. On 8 FML-bench-Lite tasks, adding IdeaCompass to Autoresearch and AdaptiveSearch improves benchmark-level NI and win rate, and the two augmented hosts rank first and second among 9 agents.

Critical strengths
  1. A well-motivated problem and an elegant representation. Making exploration in research agents measurable is a useful direction. Decoupling direction (angle) from magnitude (norm) in an unnormalized embedding is simple, interpretable, and backed by matching contrastive and ranking losses. The motivating example (Table 1) clearly shows a failure mode of general-purpose embeddings.
  2. Lightweight and plug-in. The method attaches to existing hosts without additional validation executions and adds only about 6–8% API cost.
Critical weaknesses
  1. The method measures where the search has been, not where it should go, and its core premise is not validated. (a) The embedding is trained without any experimental outcomes (Appendix C), so its distances reflect how different two edits are under a coarse 10-category / 5-level taxonomy, not whether a direction is promising. (b) The stall itself is detected by a standard improvement threshold (Gt≤ϵG_t \le \epsilon) that does not involve the embedding. The embedding only chooses between “diverge” and “converge”, and then picks one of K=2 candidates by distance. Even with a perfect diagnosis, this is a coarse switch rather than guidance on where to search next, which is the paper’s stated goal. (c) The causal chain “stagnation is caused by over-concentration or over-dispersion, and correcting spread restores progress” is never tested directly. In Table 4, the post-intervention gains are on the order of 0.002–0.016, with no variance and no episode counts. Moreover, the larger spread change after Collapse largely follows from the selection rule itself (argmax distance), so it is not independent evidence.
  2. The empirical gains are concentrated in a single task, and the method’s configuration appears tuned on that task. (a) Table 6 shows that DomainBed-CM alone accounts for about +0.0159 of AdaptiveSearch’s +0.0161 average NI gain; the other seven tasks net roughly zero, with drops on PrivacyMeter and ART. For Autoresearch, DomainBed-CM accounts for about +0.0278 of +0.0338 (~82%). The mean-of-3 gains are comparable to the standard deviations (±0.039 and ±0.035) with only 3 repetitions. (b) All ablations are run on DomainBed-CM. Performance is highly non-monotonic in the thresholds: 0.20/0.80 gives 0.498, while 0.05/0.95 and 0.30/0.70 fall to 0.09–0.10, below the host without IdeaCompass (0.183). The default thresholds are the best setting on the task where the gains are largest. Running the same controller with text-embedding-3-large (0.067) is also far worse than the bare host, which suggests the mechanism is fragile and can actively hurt search. On the Qwen3.8-Flash backbone, there is no effect at all.
Questions for authors
  1. If the controller performs its diagnosis from LLM-predicted direction and magnitude labels (from the same annotator used to build the training data) instead of the learned embedding, how does performance compare? If it is similar, what does the learned representation add?
  2. Can you report results with thresholds selected on held-out tasks, or ablations on tasks other than DomainBed-CM? Can you also report the benchmark-level gains excluding DomainBed-CM, together with significance across repetitions?
  3. Does the K=1 “none” condition in Table 5 include the injected diagnosis in the prompt? Please separate the contribution of the prompt injection from that of candidate selection, and report results for K>2.
  4. Has any human validation been done of the Claude Sonnet 5 labels (agreement rate, error analysis)? Since the direction AUROC is evaluated on the same LLM-defined taxonomy the model is trained on, how should the comparison with general-purpose embeddings be interpreted?
  5. (Clarification) The representation data includes all 8 evaluation tasks and is split by run rather than by task. Do the downstream results change if the encoder is trained on task-disjoint data? Why does the magnitude loss use the 10-level scope × novelty ordering while the evaluation uses the 5-level scope?
Reviewer_NWx5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper learns an ML-specific embedding in which angular direction represents the type of modification and vector norm represents its scope, thereby turning executed ideas into a measurable trajectory. I really like the research question. However, the methodology is rather crude, and the experimental evaluation lacks the rigor needed to support the paper's central claims.

Critical strengths

The paper addresses a fundamental challenge in autonomous research: how to measure past exploration and use this information to guide future research decisions. I particularly appreciate the attempt to move beyond unstructured research histories toward quantitative exploration modeling.

Critical weaknesses

Lack of independent validation of the learned representations. The authors use Claude Sonnet 5 to generate direction and magnitude annotations, then evaluate the learned embeddings against the same annotation framework. This creates a circular evaluation: the reported AUROC and Spearman correlation primarily measure agreement with LLM-generated labels rather than whether the embeddings capture meaningful relationships between scientific ideas. Surprisingly, the paper includes no human evaluation, not even a basic assessment of annotation quality.

Serious concerns about task-level data contamination. The embedding training data comes from 18 FML-bench tasks, including all eight FML-bench-Lite tasks used for downstream evaluation. Although experimental outcomes are excluded from representation training, the encoder has already been exposed to research ideas from the evaluation tasks. This substantially weakens the generalization claims. A task-disjoint evaluation is essential to establish whether IdeaCompass works on genuinely unseen research problems.

Questions for authors

Why is there no human evaluation of the direction and magnitude annotations? Can the authors provide an independently human-annotated test set to validate both the labels and the learned embedding geometry?

Why were trajectories from all eight downstream evaluation tasks included in representation training? Can the authors retrain the encoder using strictly task-disjoint data and report performance on unseen tasks?

What evidence supports interpreting low embedding spread as harmful collapse and high spread as harmful dispersion? Can human experts independently evaluate whether the detected states actually represent inappropriate research behavior?

How does IdeaCompass compare with simpler approaches that use the ten direction categories directly, or ask an LLM to assess exploration diversity and recommend the next research direction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.