- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces Open-Endedness Bench (OEB), a benchmark-agnostic framework for evaluating the epistemic process of research agents from their execution records rather than from task outcomes or reference answers. OEB converts heterogeneous agent logs into a unified epistemic event graph, where propositions stated by the agent are linked to executed actions and observations.
I find the motivation timely and the execution-record perspective interesting. However, I have concerns about whether the proposed scores are sufficiently validated as measures of “epistemic competence,” and how much practical value the resulting scores provide beyond descriptive trajectory analysis.
Critical strengths
1. The paper targets an important limitation of outcome-only agent evaluation.
For open-ended research tasks, a final score alone cannot distinguish disciplined experimentation from accidental success, and OEB directly targets this gap.
2. The separation between propositions and executed actions is well motivated.
The event-graph formulation avoids rewarding unsupported narrative reasoning by requiring credited propositions to be connected to actual executed actions. :chatgpt-content-reference{index="2"}
3. The framework is relatively general across heterogeneous research-agent settings.
The same graph construction and measurement layer are applied to post-training, chip design, and training-speed optimization, with benchmark-specific processing largely isolated to the log-conversion stage.
Critical weaknesses
1. The link between the proposed scores and epistemic competence is not fully convincing.
The axes are built from hand-designed opportunity-based rates, but it is not clear that a higher score always means better research behavior. For example, E2 favors experiments that resolve explicitly stated propositions, which may penalize useful exploratory experiments that are not framed this way in the trajectory.
2. The framework is only partially benchmark-agnostic.
While the scoring layer is shared, each benchmark still needs a custom log converter and a world manifest specifying task rules and setup. These choices may affect what is counted as an environment action, a violation, or an evaluation opportunity.
3. The validation of E1–E3 is relatively weak.
The paper shows that scores are stable under repeated judging and validates E4 against external benchmark audits, but there is no comparable external validation for E1–E3. Consistency across judges does not by itself show that these scores measure the intended epistemic properties.
4. Some of the empirical conclusions are mainly correlational.
For example, better runs tend to explore more new ideas in the second half, but this could also reflect stronger models, better stopping behavior, or task differences. The current analysis does not show that encouraging more late-stage exploration would itself improve performance.
5. The evaluation depends heavily on what the agent writes into the trajectory.
For E1–E3, missing reasoning is difficult to distinguish from reasoning that never happened. Since different models expose very different amounts and styles of intermediate reasoning, this may complicate cross-model comparisons.
6. The practical value of the resulting scores is unclear.
The paper provides several competence dimensions and persona traits, but it does not yet show how these measurements would help with model selection, agent design, or intervention beyond simpler outcome scores and trajectory statistics.
Questions for authors
Q1. How do the authors validate that E1–E3 actually measure epistemic competence?
The current experiments mainly show that these scores are reproducible across repeated judging. It would be useful to compare them with expert judgments of research quality or with downstream research success.
Q2. How sensitive are the results to the benchmark-specific converter and world manifest?
Since these components define the task setup and evaluation rules, it would help to quantify how much the final OEB scores change under reasonable alternative conversions or manifests for the same benchmark.
Q3. Can the authors provide stronger external validation for E1–E3, similar to the auditor-based validation for E4?
At present, E4 has a clear external reference, while the other competence axes are mostly validated through scoring stability.
Q4. Does the late-stage exploration result remain after controlling for model and task effects?
The current correlation is unclear whether late exploration itself is beneficial or simply associated with stronger models or easier tasks.
Q5. How should OEB compare models with very different reasoning verbosity?
Since E1–E3 depend on propositions written into the trajectory, terse models may generate fewer opportunities than verbose models. Could the authors test whether the conclusions remain stable after controlling for reasoning length or by prompting the same model to produce more or less explicit reasoning?
Q6. What practical decision is OEB intended to support?
For example, can OEB help select a better research agent, identify trajectories that should be trusted, or guide interventions that improve future performance beyond what can already be obtained from outcome scores and simple trajectory statistics?
Thank you for the review. We answer the questions in order.
Q1 / Q3. External validation of E1–E3. We ran a blinded human audit (general response §1). Two independent annotators labelled 80 randomly sampled items and agree with the judge on 74% and 73% of them. On downstream success: within a task, the correlations of E1–E3 with the outcome are close to zero (−0.02, −0.12, +0.14; every interval includes zero). For E4 it is −0.14 [−0.23, −0.02], because runs that hack score higher. On these short tasks the outcome does not tell us how an agent handled its evidence, which is what E1–E3 are meant to measure (§6).
Q2. Sensitivity to the converter and the manifest. For the manifest, two fields affect E4. Without the house rules, E4 still catches 10 of 11 cheating runs, and the clean runs it flags drop from 13 to 5. With and without the declared size of the evaluation set, E4 changes in 3 of 91 PostTrainBench runs. For the converter, the choice that matters most is which text enters the record. Visible messages and actions enter for every model, and hidden reasoning enters for none. Action and observation text is copied verbatim. For each run the converter also records a count of what it could not carry over (for example, file edits whose content the log does not contain).
Q4. Late exploration with controls. Ranking within a task already removes task effects. With model fixed effects the slope goes from +0.54 to +0.40 (95% CI [0.16, 0.67]). Adding run length and the number of research experiments, which capture when an agent stops, gives +0.37 [0.04, 0.69]. When the same model ran a task twice, the higher-scoring run tested more late ideas in 11 of 16 pairs. We did not intervene on exploration, so the paper now calls this an association (§6.2).
Q5. Models with different verbosity. We ran this control on the 91 PostTrainBench runs. We removed task effects and regressed each score on the amount of visible text per step (log scale).
The differences between models in E1–E4 remain when models write the same amount, and the ordering of models is preserved. T6 describes how an agent writes its account, so we expect it to move with the amount of text. Prompting the same model to write more or less would need new runs, and we leave that to future work.
Q6. Practical use. We see three uses that an outcome score does not offer. First, it shows which claimed improvements to trust: agents often call a result an improvement when it is not, and then build on it (Table 2). Second, it points to the specific actions where hacking happened, with the rule and the quoted evidence, so an auditor can check them (E4, Table 3). Third, it informs harness design: on Chip-Bench, the fourth and later tries of an idea make up 36% of experiments and bring 10% of the gain, which argues for breadth (Fig. 4).
W1. Does E2 miss exploratory experiments? Experiment yield counts every research launch whose result the agent reads and judges, whether or not it stated a proposition beforehand. The only launch without credit is one whose result the agent never interprets in words.
W2. Is the method benchmark-agnostic? The extraction prompts, the jury questions, the graph and every scoring rule are identical across the three benchmarks. Two things are written per benchmark: the converter, which copies log text into a common schema, and the manifest, which states the task's own rules.
Presentation. We took this rating seriously and made four changes. §3 now follows a single example through extraction, linking and scoring. §4.2 states how each rule treats engineering checks, withdrawn results and stated causes. §2 states what the record contains and what it leaves out. §6 reports the late-exploration result as an association, next to its controls.