acceptodds
Withdrawn from ICLR 2027

You Don't Need to Read Papers Anymore: A Provenance-Disciplined Advisory Tier for Autonomous Research

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autonomous research agents that work alone fail in a characteristic way: one context holds every result and the story built on it, so each new result is read through that story and the agent checks its own hypotheses against its own framing of the evidence. In practice the agent clings to its first direction until outside feedback, usually a human's, arrives: it takes run after run to leave an approach that is failing, and keeps working around problems whose solution the literature already holds. We propose separating execution from epistemic control by layering knowledge by generality: a task tier that runs the experiments, a domain-supervisor tier that owns the domain literature, and an advisory tier that answers only from general ML knowledge. YoDA (You Don't need to read papers Anymore) implements the advisory tier as an agent outside the experiment loop. It shares neither context nor success criterion with its asker, receives only questions the supervisor has abstracted to general deep learning, answers each in a disposable per-thread agent, and marks every claim as corpus-grounded or not. Its substrate is a literature knowledge base consulted rather than browsed: 769 papers digested at registration into fixed-schema notes whose quantitative claims carry fixed provenance, under a contamination rule that never admits an asker's claim as evidence and a reading-priority score that corpus growth cannot move. On MLE-bench (22 competitions, private grading, one base model in every tier), the three-tier pipeline took 17 of 22 medals within one hour, the most of five pipelines, and 19 by 24 hours with each cell stopped at its first privately graded medal; the cell-level records show the supervisor tier producing the largest single gains and catching defects the executor could not see, and the executor overturning the supervisor's wrong priors by measurement.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
autonomous research agents, LLM agents, multi-agent systems, epistemic control, provenance, literature knowledge base

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Pc6pOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that an AI research agent working alone gets stuck on its first idea because it judges every new result against the story it has already built. To fix this, it splits the work across three agents: an executor that runs experiments, a supervisor that directs it, and an advisor (YoDA) that answers general ML questions without seeing the task details. The advisor answers from a library of 769 summarized papers and marks which parts of each answer come from those papers. On MLE-bench Low (22 Kaggle competitions, Claude Opus 5, one run), the system wins 17 medals in one hour, against 16 for each of two baselines.

Critical strengths

A simple idea, plus a useful side finding. Putting a reviewer agent outside the agent's own context is easy to adopt, and having every answer mark what comes from the papers makes the advice easy to check. The most useful part of the paper, though, is the transcript analysis. Supervisor guidance failed mostly because agents only see messages between turns, so directions often arrived after the work was done.

Critical weaknesses
  1. The paper doesn't show that the problem exists, and the solution is too heavy for what it gains.
  • "Validation collapse" is never measured in the paper's own runs. The paper's own transcripts point to a different problem: late messages (Sec. 5.2).
  • The paper library may not fit the benchmark. It is mostly about making models smaller and faster, such as quantization and pruning (Sec. 4), while Kaggle tasks need everyday skills like feature engineering and ensembling that frontier models already have. So even when the system does well, we can't tell whether the papers helped at all. The paper also doesn't say how often the advisor was consulted during MLE-bench, or what it changed.
  • There are no cost numbers. One consultation costs 48k–70k tokens (App. D), but no pipeline has a per-competition cost. Matching the baselines at lower cost would count in its favor; without numbers, it looks like extra cost for no clear gain.
  1. The MLE-bench results don't support the claims.
  • Everything is one seed, and the authors admit the 17 vs. 16 gap is within noise.
  • The pipelines ran under different setups (App. A.1). The full system used an earlier setup with a 20-minute training budget. The ablations and AIDE shared a GPU and started up to five minutes early. AIBuildAI had the GPU to itself and stopped after 34 minutes on average.
  • The 19 medals come from the stage with test-set feedback, yet they appear in the abstract and contributions.
Questions for authors
  1. How often was the advisor consulted in each competition, and did it change any outcome?
  2. What did each pipeline cost per competition?
  3. Can you rerun all pipelines under the same setup, with at least three seeds and no test-set feedback?
  4. How does the system compare with a fresh-context critic, the same advisor with plain retrieval, and a two-tier system with message delivery fixed?
  5. Can you show the problem directly? For example, count how often a solo agent keeps trying an idea that isn't working, and check whether that happens less with the advisor.
Reviewer_RPpfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper aims to address a question that the research AI agents that work alone often get stuck on their first idea and judge their own results by their own stories, and they missed fixes already known in the literature.

The paper proposes splitting the agent like a research lab which includes executors that run the experiments, domain supervisor that knows the topic's paper and directs the experiments, YoDA advisor that answers only general questions but never sees the details like the scores.

Some interesting designs are: 1. each question to the advisor is answered on the fly by a throwaway helper agent 2. every claim in an answer is labeled either "backed by the paper library" or "not backed ~"

The paper also presents a paper library of 700+ papers on model optimization

On MLE bench, the system wins 17/22 medals with opus 5 as the backbone model.

Critical strengths

The idea is intuitive and easy to follow. The authors clearly spent effort searching existing related works

Critical weaknesses

I am mainly concerned that the results might not be significant enough since the margin is one medal and it comes from one seed without any significance test. Figure 3 says the YoDA rows “come from earlier-protocol arms” so YoDA and the baselines did not run under the same protocol based on this description which makes the comparison unfair. The core claim seems untested given that the conclusion says "Whether the advisor changes outcomes is the next measurement"

I am also concerned about the writing reads defending, makes a reader wonders about the motivation behind. For example: "...position, not a role label or a trained head”, in which “trained head” is never introduced, so the reader wonders where it came from. This example also leads to another concern which is the sudden or unsupported materials. For example, at the end of 5.1, with nothing leading up to it: “The domain-specific knowledge dissemination system demonstrated rapid convergence.” Also, Section 4.4 introduces a citation score formula with fixed constants, but it is never linked to the results. “Earlier protocol”, “earlier arms” and “fixed-budget arm” are not explained in the main text.

And some terms are unusual in this field throughout the paper like: “epistemic control”, “provenance discipline”, “quarantine”, “contamination rule”

Questions for authors
  1. to what extent are the authors willing to expand their experiments scope including running on more benchmarks and ablations on different models and other parameters
  2. How would you measure whether the margin comes from the system design or chance?
Reviewer_ri7VOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes YoDA, a literature-based advisor for autonomous research agents. It separates experiment execution, domain supervision, and general methodological advice into three tiers. The advisor answers questions in fresh conversations using structured notes covering 769 papers, labels claims according to their support in the corpus, and restricts how new information enters the knowledge base. Validation consists of MLE-bench experiments, comparisons with alternative agent configurations, and an illustrative advisory interaction.

Critical strengths

I can relate to the motivation: a research agent can benefit from advice that challenges its current interpretation and connects its problem to relevant literature. The paper implements a concrete system around this idea. Its reported 17/22 medals on MLE-bench Low demonstrate useful absolute performance, and rerunning AIDE and AIBuildAI with the same base model is a positive aspect of the evaluation. The deployment observations also identify practical coordination problems that may be useful to researchers building similar systems.

Critical weaknesses
  1. The paper does not establish the effectiveness of its central contribution.

The motivation for an independent advisor is reasonable, but the evidence does not establish that YoDA improves research outcomes. Section 3.1 provides an illustrative interaction without a controlled comparison. Much of Section 5.2 discusses contributions from the supervisor in runs where the advisor was absent. The reported one-hour advantage is also small: 17 medals versus 16 for AIDE and AIBuildAI, with one seed per competition; AIDE obtains more golds.

The authors explicitly leave whether the advisor changes outcomes to a future measurement. I appreciate this acknowledgment, but this is the central question the present submission needs to answer. The current results establish a functioning research-agent pipeline more clearly than they establish the advisor’s contribution.

  1. The proposed reliability mechanisms are insufficiently evaluated.

The notes are generated from titles and abstracts together with the note-writing model’s general knowledge. An answer can therefore be labeled grounded because it is supported by generated synthesis in a note, without establishing that the underlying paper supports the claim. This is consequential for method details, limitations, and practical recommendations.

The manuscript acknowledges this distinction, but does not report quantitative fidelity findings sufficient to assess its consequences. Likewise, the example in Section 3.1 does not establish that context separation caused the successful correction. Access to the relevant note or an explicit checking instruction could also explain it. Evaluations of attribution fidelity, isolation, labels, and retrieval alternatives are deferred to Section 6. These tests concern the proposed contribution itself.

  1. The distinction from prior systems is not sufficiently established.

The related-work section broadly describes earlier frameworks as keeping “one memory and one objective.” This conflates shared storage, agent contexts, and evaluation criteria. MetaGPT includes role-specific message filtering, and AutoGen supports configurable agents and interaction patterns. Shared system memory in co-scientist does not by itself establish identical contexts or evaluation responsibilities across agents.

An integration of existing ideas can be valuable. Here, however, the architectural distinction needs a more precise comparison, and the experiments do not establish why the particular integration is effective.

  1. The presentation substantially obstructs assessment.

I found the entirety of the manuscript difficult to follow despite careful reading and it feels very rushed. The manuscript repeatedly uses compressed terminology before explaining the operations, relies heavily on the lab analogy, and interleaves architectural descriptions with references to later examples and claims about effectiveness. The reader has to reconstruct both the mechanism and the argument. This makes the contribution and its supporting evidence unnecessarily difficult to assess. I understand the authors have disclosed the usage of AI tools, but acknowledging the tools doesn't justify leaving the generated manuscript unchecked for fluency, coherency, and readability. The current manuscript is nowhere near the standard for a conference submission.

  1. The experimental protocols and reporting further weaken the comparisons.

The ablation is internally inconsistent. Section 5.2 reports 17/16/15 medals, while Figure 4 shows 19/16/15 and labels the comparison as 24 hours. Appendix A.1 specifies a one-hour ablation budget, whereas Table 3 labels the merged and two-tier results as 24 hours.

Appendix A.1 also states that the full-system results come from earlier runs under a different protocol. Training allowances, GPU allocation, and preparation differ across conditions. This prevents the merged-agent comparison from cleanly isolating context separation. Removing the advisor also does not distinguish advisory interaction from access to its corpus. A comparison giving the supervisor the same literature access would help resolve that attribution.

Finally, the extended full-system result uses private-test-dependent stopping, which the authors acknowledge supplies test feedback. The 19-medal result therefore cannot establish superiority under a protocol without such feedback. These qualifications do not erase the system’s absolute performance, but they substantially limit the conclusions supported by the comparisons.

Questions for authors
  1. In the full-system MLE-bench runs, how often was YoDA consulted, what advice did it provide, and which subsequent decisions changed? The current analysis mainly identifies contributions from the supervisor. Evidence from the recorded interactions would help establish the advisor’s actual role.

  2. What were the sample size, checking procedure, and findings of the stated note-quality audits? In particular, did the checks assess whether generated method details, limitations, and recommendations were supported by the underlying papers, beyond consistency with their abstracts? This would help assess what the grounding labels establish in practice.

  3. Could the authors reconcile the ablation results and durations? Section 5.2 reports 17/16/15 medals, Figure 4 shows 19/16/15 at 24 hours, Appendix A.1 specifies a one-hour ablation budget, and Table 3 labels the merged and two-tier results as 24 hours. Please clarify the wall-clock budget, training allowance, GPU allocation, and stopping rule applicable to each condition so the comparisons can be interpreted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.