- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes IdeaCompass, a framework designed to make exploration measurable and steerable for autonomous ML research agents. IdeaCompass introduces an ML-specific idea embedding model (fine-tuned from gte-modernbert-base) where the vector direction represents the type of ML modification (10 categories, e.g., optimizer, model architecture, loss function) and the vector norm represents modification magnitude (5 ordinal scopes).
Using this embedding space, the agent's history is considered as a trajectory. A controller dettects search stagnation: when validation improvement stalls over a window W, it checks the trajectory spread, and if the spread is low, it diagnoses Collapse and steers the agent towards diversification; if the spread is high, it diagnoses Dispersion and steers towards consolidation.
Critical strengths
- This paper has an interesting geometric framing of search dynamics. Formulating exploration trajectory in terms of decoupled direction (what changes) and magnitude (how much changes) is an intuitive idea and can be conceptually sound.
- Distinguishing between stagnation caused by over-concentration (collapse) and stagnation caused by aimless jumping (dispersion) provides a more nuanced state diagnosis than simple no-progress counters.
- The paper provides episode-level analyses (Table 4) examining how spread dynamics and episode durations shift under collapse and dispersion interventions.
Critical weaknesses
On potential data leakage,
- In appendix C lines 648-654, "The representation dataset is collected from 18 FML-bench tasks, including the eight FML-bench-Lite tasks used for downstream evaluation; the representation split is therefore run-disjoint rather than task-disjoint.".
- The embedding encoder and direction prediction heads are trained on idea proposals collected from the exact same 8 benchmark tasks used for downstream evaluation.
- Lack of OOD evaluations.
On outlier-driver gains
- A closer inspection to Table 6 in Appendix B suggests that the performance improvement is mainly from one single task DomainBed-CM, and improvements on other 7 tasks are marginal or negative.
- The ablation experiments are conducted entirely on the DomainBed-CM task and raises the concern of cherry picking analysis.
Questions for authors
Refer to the above #weaknesses