acceptodds
Under review as a conference paper at ICLR 2027

The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

Abstract

LLM agents increasingly work on long-horizon tasks, and the decisions they make along the way, such as which hypothesis to test or which implementation to build on, determine the outcome of the whole run. Making these decisions well is becoming a key capability for both engineering and research agents. We refer to the ability to make good long-horizon decisions as the taste of an agent. While existing benchmarks measure the end-to-end success of agents on long-horizon tasks, none of them measures the taste of an agent. To address this problem, we build Taste-Bench, a benchmark of taste questions constructed automatically from trajectories that agents produced in engineering and research tasks. Each question presents a decision fork, a point in a trajectory where multiple directions are available and one of them leads to a better outcome, and the evaluated model chooses among these directions without seeing what happens after the fork. We mine these forks automatically from parallel attempts at the same task and from detours inside a single trajectory, without needing human annotation. We evaluate frontier models on Taste-Bench and find that the best model answers only 59.7% of the questions correctly. We further find that forks whose deciding evidence appears later in the trajectory are much harder for every model, and that a larger reasoning budget does not improve the accuracy. Finally, we show that taste can be trained. We distill the judgment of a teacher that has seen the outcome into a student model, and the student makes better decisions on unseen tasks and improves end-to-end success on held-out SWE-bench Pro tasks.

Primary area:
datasets and benchmarks
Keywords:
long-horizon agents, benchmark, LLM agents, decision making, judgment, distillation, software engineering agents, ML research agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M3XvOpenReview β†—
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper defines "taste" as the ability of LLM to choose the solution path that leads to the better outcome in long horizon. It builds TASTE-BENCH, 502 binary-choice questioned automatically mined from trajectoreies on SWE-bench, using parallel or detour trajectories. Then the paper shows a number of analyses on the challengingness of the dataset, and how models behave, and some distillation results on how we can use the dataset to further improve models.

Critical strengths
  • carefully designed annotation-free construction of a dataset
  • transparent evaluations
Critical weaknesses
  • I don't see how this is different from RL. Picking the better option at a form is basically estimating which action has the higher Q-value, and the labels come from single rollouts, which is what RL methods and process reward models already do. The paper doesn't explain what this captures that RL can't, and there's no comparison with RL, PRM, or step-level DPO trained on the same data. (This is the most severe issue of this paper)
  • The labels may not really reflect the decision at the fork, because the two attempts don't actually share the "identical" prefix in parallel forks (it says it follows success trajectories' ones), and good option was only tried after the agent saw the failure in detour forks. This should have been made clear and double checked empirically whether this doesn't make much implications. While human judgements were used to assess the quality of the dataset, I do not agree with the arguments, as the human reviewers have observed the outcome. So it is merely a sanity check whether there were no systematic errors that checks wrong outputs as the right output.
  • In experiment figure 3, while it is argued that the state of the art models have poor taste of 59.7%, we do not know whether it is because of poor taste or it is because of bad dataset that no one can guess. Upper bound results should have been provided, e.g., human accuracy. The question generator is also the same as the top-ranked model, so the results may also have been biased.
  • The end-to-end experiment in distillation experiment does not isloate student's own contribution. The advice is written from earlier runs on the same task with knowledge of the hidden test results, so even wrong advice may give the agent useful hints. a few more baselines needed to clarify.
Questions for authors
  1. How is this different from training a process reward model or doing RL on the same rollouts? Could you add a baselines like PRM or step-level DPO? Why can't 'taste' learnt from usual RL training simply from SWE-bench?
  2. What is the human accuracy before seeing the outcome?
  3. Does the ranking change if you use a different model as the generator?
  4. In the end-to-end experiment, what happens with random advice, or with both options shown without a recommendation?
3 comments
AuthorsOpenReview β†—

Response to Reviewer M3Xv (1/3)

We thank the reviewer for the thoughtful and constructive feedback. We are encouraged that the reviewer finds our dataset construction carefully designed and annotation-free, and our evaluations transparent. Following your questions, we conducted several new analyses and revised the paper accordingly. We carefully address your concerns point by point below.


W1 / Q1: How is this different from RL / PRM / step-level DPO? Why can't taste be learned from RL on SWE-bench?

Thank you for raising this central question. We understand your concern to be: if choosing the better option at a fork is equivalent to estimating which action has the higher value, what does Taste-Bench capture that RL, PRMs or step-level DPO do not?

We would like to clarify that Taste-Bench is a benchmark rather than a training method. PRMs, value functions and RL policies are possible ways to make the decisions in our questions, and Taste-Bench provides outcome-labeled test items, with the full decision-time context, on which any of these estimators can be evaluated. In this sense, our work is complementary to RL and PRMs rather than an alternative to them.

To examine whether the benchmark can be solved by evaluating each step in isolation, we compared our standard protocol, where the model sees both candidates and chooses one, with pointwise scoring in the style of a PRM, where the model sees one candidate at a time and outputs the probability that the agent succeeds after taking it (150-question subset).

ModelStandard protocolPointwise scoringQuestions with tied scores
GPT-5.6 Sol62.760.78.7%
GPT-5.6 Terra56.750.713.3%
GLM-5.263.352.714.7%

Pointwise scoring does not improve accuracy for any of the three models, and it often assigns the same score to both candidates. We have added this comparison to the revision πŸ“[Sec. 4.2, L267–L268], πŸ“[App. D.4].

In the original manuscript, two further results show that Taste-Bench measures long-horizon judgment rather than the evaluation of a single step:

Time horizon of the deciding evidenceMean accuracy of 14 models
Already visible in the prefix62.3
Only visible after substantial later work21.0

Accuracy falls monotonically as the deciding evidence appears further in the future, close to the 25% of random guessing at the longest horizon (Section 4.3). In addition, a larger reasoning budget does not help: a 17-fold increase in reasoning tokens changes the accuracy of GPT-5.6 Sol by only βˆ’0.2 points (Section 4.4). These findings suggest that the questions require anticipating the consequences of a direction, which a model cannot obtain by evaluating the current step more carefully.

Regarding RL on SWE-bench, an episode-level reward does not identify which of the many decisions in a long trajectory caused the failure. Our mining step turns such trajectories into decision-level supervision, two candidates at one state with an outcome label, and these items can serve as training and evaluation data for RL and PRM methods in future work.

To verify this, we also ran a pilot that trains the same base model directly on the binary fork label, using 215 parallel engineering questions from an earlier version of the question pool and two task-disjoint folds:

Training method (accuracy on held-out tasks)After trainingBefore training
GRPO with the fork outcome as reward35.336.7
Rejection-sampling fine-tuning26.937.0

Neither method improves held-out accuracy over the base model. This pilot motivates our choice to distill the reasoning of a teacher model that knows the correct answer, instead of fitting the label πŸ“[App. G].

AuthorsOpenReview β†—

Response to Reviewer M3Xv (2/3)

W2: The labels may not reflect the decision at the fork, and the human reviewers observed the outcome.

Thank you for pointing this out. For detour forks, the generator rejects any fork whose better direction can only be named after seeing the failure, and it checks that the prefix does not reveal the failure or the later fix πŸ“[L150]. For parallel forks, the two attempts share the same task and diverge at the fork, and a question is released only when all four judges confirm the label from the full record. Regarding the human review, we agree that judging after seeing the outcome only checks the labels. We therefore also report the human accuracy before the outcome is shown, which we describe in our next response (W3 / Q2).

To test whether the label depends on which attempt provides the history, we also re-evaluated all 172 parallel questions with the prefix taken from the failing attempt instead of the passing one:

ModelHistory from the passing attempt (paper)History from the failing attemptQuestions with the same answer
GPT-5.6 Sol70.358.779%
GLM-5.263.455.274%

With either history, both models remain well above the 25% of random guessing, and most answers are unchanged πŸ“[App. D.5].


W3 / Q2 / Q3: Is 59.7% low because of poor taste or because the dataset is unanswerable? What is the human accuracy before seeing the outcome? Does the ranking change with a different generator?

Thank you for this important question. We understand your concern to be: without a reference point, a low score could reflect noisy labels rather than limited taste.

We first note that our headline metric counts a question as correct only when the model chooses the better direction in both candidate orders. Random guessing therefore scores 25% rather than 50%, and a model that always prefers the same position scores 0%. On this scale, 59.7% is well above chance, and the remaining question is how far it is from what a human can achieve.

Our two-stage human review helps to answer this question. Each reviewer first chooses a direction before any continuation or outcome is shown, and only then sees the recorded outcomes and judges which decision was better. The second stage measures label quality, and the first stage gives a human reference at decision time.

Label quality (after seeing the outcome)Result
Human judgments that support the mined label98.8% (170 of 172)
Agreement between the two reviewersCohen's ΞΊ = 0.97
Accuracy at decision time (68 shared questions)Accuracy
Human reviewer 175.0
Human reviewer 257.4
Best model (GPT-5.5)66.2
Median of 14 models55.9

The mined labels are almost always confirmed once the outcome is known, and the more experienced reviewer outperforms every evaluated model before the outcome is known. Together, these results indicate that the labels are reliable and that the 59.7% of the best model reflects the difficulty of the decisions rather than noise in the dataset. We have added this human reference to the revision πŸ“[L265–L267], πŸ“[App. F.2].

Regarding the generator, GPT-5.6 Sol is statistically tied with GPT-5.5 on Taste-Bench, the two smaller models of the same family rank 5th and 8th, and the generator is excluded from the judge panel that filters the questions. Re-building the benchmark with another generator is beyond the rebuttal period, and we will add a generator-swap study on a subset in the final version.

AuthorsOpenReview β†—

Response to Reviewer M3Xv (3/3)

W4 / Q4: The end-to-end experiment does not isolate the student's contribution. What happens with random advice, or with both options shown without a recommendation?

Thank you for this insightful point. We agree that the notes themselves carry information about the task. We understand your concern to be: the end-to-end gain could come from the task information in the notes rather than from the judgment of the student.

The contribution of the student is its judgment at each decision point. To isolate it, we compared the student with the untrained base model given exactly the same fork questions, prompt and decision rule on the 98 held-out forks:

AdvisorDecision points where the advisor picks the better direction
Untrained base model55.1%
Distilled student78.6% (p = 0.0015, 18 tasks better and 3 worse)

Since both advisors receive the same historical information, this comparison separates the distilled judgment from the information in the notes. The student is never trained on these tasks, because training and evaluation use disjoint sets of tasks.

We also ran the controls you suggested. On the paper's executor, we compare different advisors on the same 41 held-out tasks:

Advice given to the executor agentTask success
No advice14.6%
Untrained base model30.8%
Random advice34.1%
Distilled student33.7%
Correct advice at every fork39.0%

In addition, we ran notes that show both options without a recommendation on a separately deployed copy of the same executor agent. Since this deployment differs from the paper, we compare each type of notes only with the run without notes on the same deployment:

Notes given to the executor agentChange in task success over no notes (percentage points)
Note header and submission checklist only, without decision points+0.0
Decision points with both options, without a recommendation+10.5
Decision points recommending the worse option+10.5
Decision points recommending the better option+15.8

These controls show that the decision points in the notes carry useful information about the task, which is why we use the fork-level comparison above to isolate the judgment of the student.

In the original manuscript, we present the end-to-end success rate as the main evidence for the student. In the revision, we use this fork-level accuracy as the primary measure of the student's contribution, present the end-to-end results as supporting evidence, and revise Finding 6 accordingly πŸ“[L442–L443], πŸ“[App. H].

We hope these clarifications and the additional evidence address your concerns. Please let us know if you have any further questions!

Reviewer_EdMoOpenReview β†—
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the TASTE-BENCH benchmark, designed to measure a long-term agents’ ability to make better decisions when faced with multiple possible directions during execution. The authors define this ability as "taste." The paper digs into historical execution traces of software engineering and AI research tasks to identify two types of decision branches: one coming from parallel trajectories with different outcomes for the same task, and the other from correction trajectories where an agent abandons a wrong path and eventually recovers in a single run. After automatic filtering and manual verification, they built a benchmark with 502 problems and used a reverse option order evaluation protocol for fair testing. Experiments on 14 models show that the best-performing GPT-5.6 Sol achieved a strict accuracy of 59.7%, suggesting that current models still struggle to consistently judge which path is better at the point of decision. Furthermore, the authors used teacher reasoning distillation to train student models, improving their judgment on unseen tasks, and in end-to-end experiments with SWE-bench Pro, increased the fixed execution model's task success rate from 14.6% to 33.7%. Overall, this paper attempts to turn an agent's long-term decision-making ability into something that can be automatically evaluated and trained, establishing a new evaluation benchmark.

Critical strengths
  1. The research perspective is both innovative and practically meaningful. This paper starts from the common problems that arise during long-term task execution and proposes TASTE-BENCH, which is used to evaluate an agent's ability to judge the pros and cons of different execution paths at key decision points. This benchmark offers a new approach for assessing an agent’s decision quality.
  2. The paper further validates the practical application value of the proposed TASTE concept through experiments. The authors generated decision suggestions through distillation training and improved the task success rate of a fixed execution model on SWE-bench Pro from 14.6% to 33.7%, showing that this framework not only has offline evaluation value but could also effectively improve an agent's execution performance.
Critical weaknesses

It is difficult to determine from the paper whether TASTE-BENCH evaluates a new capability, defined in the paper as "taste", or merely assesses the ability to evaluate intermediate steps (i.e., the capability of a Process Reward Model). The absence of evaluation results using existing methods (e.g., PRM-based approaches) makes it challenging to ensure exactly what additional capabilities TASTE-BENCH is testing.

Questions for authors

Q1 Can the authors distinguish whether TASTE-BENCH evaluates long-term decision-making capability, as opposed to general programming knowledge or the ability to assess intermediate steps (similar to Process Reward Models, or PRMs)? Experimentally, is it possible to compare TASTE-BENCH against existing PRM-based methods to demonstrate that long-term decision-making capability is not fully captured by those approaches?

Q2 The distilled student advice significantly boosts task success rates on SWE-bench Pro. How can the authors tell if this improvement is due to transferable decision-making ability or just task-specific information from historical trajectories? Is it possible to have an untrained base model provide advice based on the same historical information, and then compare its performance?

1 comment
AuthorsOpenReview β†—

Response to Reviewer EdMo

We thank the reviewer for the thoughtful and constructive feedback. We are encouraged that the reviewer finds our research perspective innovative and practically meaningful, and recognizes that Taste-Bench offers a new approach for assessing an agent's decision quality. Following your questions, we conducted new analyses and revised the paper accordingly. We carefully address your questions point by point below.


Q1: Does Taste-Bench evaluate long-term decision making, as opposed to general programming knowledge or the ability to assess intermediate steps (PRMs)? Can it be compared with PRM-based methods?

Thank you for this important question, which touches on what the benchmark actually measures. We address the two alternatives in turn.

Assessing intermediate steps. Following your suggestion, we compared our standard protocol, where the model sees both candidates and chooses one, with pointwise scoring in the style of a PRM, where the model sees one candidate at a time and outputs the probability that the agent succeeds after taking it (150-question subset).

ModelStandard protocolPointwise scoringQuestions with tied scores
GPT-5.6 Sol62.760.78.7%
GPT-5.6 Terra56.750.713.3%
GLM-5.263.352.714.7%

Pointwise scoring does not improve accuracy for any of the three models, and it often cannot separate the two candidates at all. We have added this comparison to the revision πŸ“[Sec. 4.2, L267–L268].

Long-horizon decision making. In the original manuscript, accuracy depends strongly on how far in the future the deciding evidence appears:

Time horizon of the deciding evidenceMean accuracy of 14 models
Already visible in the prefix62.3
Only visible after substantial later work21.0

At the longest horizon, accuracy falls close to the 25% of random guessing (Section 4.3), and a larger reasoning budget does not help (Section 4.4). This pattern is what we expect from long-horizon decisions and not from the evaluation of a single step.

General programming knowledge. Taste-Bench is only partly correlated with SWE-bench Verified (r = +0.37 on the engineering subset), and the four highest models on SWE-bench Verified are within 4.0 points of each other there while they are 10.7 points apart on Taste-Bench (Section 4.5). For example, DeepSeek V4 Flash ranks 5th of 11 models on SWE-bench Verified but 10th on Taste-Bench. Removing the trajectory also lowers the accuracy of the best model from 58.2 to 53.4 (p = 0.02), which suggests that the score also depends on the agent's state, not only on knowing which option is better in general.


Q2: Is the improvement due to transferable decision-making ability or task-specific information from historical trajectories? Can an untrained base model provide advice based on the same historical information?

Thank you for proposing this control, which directly addresses the source of the improvement. We gave the untrained base model exactly the same fork questions, prompt and decision rule as the student on the 98 held-out forks:

Advisor (same information)Decision points where the advisor picks the better directionTask success with its advice
Untrained base model55.1%30.8%
Distilled student78.6% (p = 0.0015, 18 tasks better and 3 worse)33.7%

Both advisors receive the same historical information, and the student is never trained on these tasks, because training and evaluation use disjoint sets of tasks. The student also transfers on the benchmark itself: on held-out engineering questions, its accuracy rises from 30.0% to 47.9% (Section 5.2). In the revision, we report this control and use the fork-level accuracy as the primary measure of the student's contribution πŸ“[L442–L443].

We hope these clarifications and the additional evidence address your concerns. Please let us know if you have any further questions!

Reviewer_EgTgOpenReview β†—
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In the paper, the authors study agent's β€œtaste”, the ability to choose better directions in long-horizon tasks before their consequences become apparent. They introduces TASTE-BENCH, which automatically extracts decision forks from engineering and research trajectories and uses subsequent outcomes to label the better choice. They found that Frontier models struggle with these decisions, particularly when decisive evidence appears much later, and additional reasoning does not improve accuracy. The authors further show that this ability can be trained by distilling hindsight-informed teacher judgments into a student, improving both decisions on unseen tasks and end-to-end task success.

Critical strengths
  1. The authors provide an outcome-grounded, annotation-free supervision for decision quality. Most prior work on judging decisions before execution (Wen et al. 2025; Zheng et al. 2026; Baig et al. 2026) asks a model to rate standalone ideas or proposals, detached from any actual run. Process-supervision methods, meanwhile, need either humans to label every step (Lightman et al. 2024) or fresh rollouts from every step (Math-Shepherd). This paper takes a different route: it finds points where two attempts at the same task share a prefix and then diverge, and uses what actually happened afterward to decide which direction was better (Eq. 1). Each question therefore carries the agent’s full context at the moment of the decision. Because the labels are mined from trajectories that already exist (4,657 candidate forks from the authors’ SWE-bench Pro runs and METR’s public MALT transcripts), the benchmark grows with agent usage rather than with annotation budget. A human review of 100 questions found 98.8% agreement with the mined labels, and the two reviewers agreed with each other almost perfectly (ΞΊ = 0.973), which suggests the labels are reliable.

  2. This paper offers relevantly clear supervision for individual decisions. Episode relative advantage learning compares three choices from the same state: executing the proposed tool, replanning, and stopping. It uses the same policy for subsequent steps and measures each choice’s return relative to executing the tool. This deisgn provides a more direct training signal for individual decisions than a single reward for the entire trajectory. The unpaired-training ablation supports the value of these paired comparisons.

  3. The design incorporates a careful filtering mechanism and a strict scoring protocol. The generator (GPT-5.6 Sol) is separated from a four-model judge panel; evidence excerpts and event ordering are checked mechanically; questions guessable from candidate wording alone are dropped; and a fork is kept only when every judge confirms the label from the full record. The headline metric requires both candidate orders to be answered correctly, so position bias (Shi et al. 2025) is scored as non-taste rather than averaged away (chance = 25%, constant-position = 0%). Full prompts, per-stage yields, and per-cell counts make the construction auditable.

Critical weaknesses
  1. One concern is that the core novelty relative to prior evaluator-guided approaches remains unclear. Planning without Search [1] already uses an auxiliary value function to guide a fixed LLM, while ToolTree [2] incorporates evaluation before tool execution. arxiv.org Freezing the base model and training an external evaluator therefore does not, by itself, establish novelty. The authors should more clearly explain the distinct methodological contributions of relative-advantage learning and policy evolution, and provide direct comparisons with the most closely related learned controllers.

[1] Hong, Joey, Anca Dragan, and Sergey Levine. "Planning without search: Refining frontier llms with offline goal-conditioned rl." Advances in Neural Information Processing Systems 38 (2026): 10032-10056.

[2] Yang, Shuo, et al. "Tooltree: Efficient llm tool planning via dual-feedback monte carlo tree search and bidirectional pruning." International Conference on Learning Representations. Vol. 2026.

  1. Some key analytical outcomes of the paper are drawn from small, lopsided, and hard-to-reproduce sample. The headline Average weights 112 research questions equally with 390 engineering ones, the parallel-research cell comes from only 8 tasks, and most bootstrap intervals in Figure 3 overlap, so many rankings are not statistically separated. The SWE-bench Verified correlation rests on 11 points with a CI of [+0.04, +0.89], a closed generator discards 61% of candidates under its own rubric, and contamination from public SWE-bench Pro patches and MALT transcripts is never addressed. The authors should report subsets separately with task-clustered intervals, expand the research cell, and release the full candidate pool with the generator's decisions so the selection can be audited. A contamination check (e.g., whether models can reproduce gold patches, or accuracy split by task date vs. training cutoff) should be added to the limitations.

  2. The trivial filter removes a question only when all four older, weaker judges answer it from the candidate text alone, yet those judges still score 55.6–63.7% on released parallel-engineering questions with no trajectory (Table 8), and the frontier models are never evaluated without the prefix, so it is unclear how much of a 60–76% score reflects judgment over the trajectory versus the generator's wording. Because the candidates are short, post-hoc LLM-written summaries, the task is a two-way choice between descriptions rather than a score on the agent's actual actions, as PRMs and step-level agent evaluations provide (Chae et al.; Li et al.). The paper also never checks whether a model's benchmark score predicts the fork choices in its own rollouts, which the GPT-5.4/5.5 trajectory pool would allow. The authors may need to report every evaluated model's accuracy with the prefix removed (ideally tightening the filter so that candidate-only accuracy is near chance), and add a check that each model's score correlates with the rate at which its own rollouts take the supported direction at those forks.

Questions for authors

Please see the question proposed in weaknesses section.

1 comment
AuthorsOpenReview β†—

Response to Reviewer EgTg

We thank the reviewer for the detailed and constructive feedback. We are encouraged that the reviewer finds our outcome-grounded and annotation-free supervision a scalable route compared with prior work, the mined labels reliable, and our filtering and scoring protocol careful and auditable. Following your suggestions, we conducted the analyses below and revised the paper accordingly.


W1: Planning without Search / ToolTree.

Thank you for these references. We have added both works to the related work section as value-guided approaches that Taste-Bench could evaluate πŸ“[L459–L460].


W2: Small and lopsided samples, the macro average, overlapping intervals, and contamination.

Thank you for these concrete suggestions. In the original manuscript, we report the mean of the engineering and research subsets. In the revision, we also report the accuracy over all 502 questions with confidence intervals that resample whole tasks, and a paired test against the top model πŸ“[App. D.2].

ModelPaper metricAll 502 questions [95% CI]p vs. GPT-5.6 Sol
GPT-5.6 Sol59.758.2 [53.5, 63.0]-
GPT-5.559.557.4 [52.2, 62.5]0.77
Grok 4.554.653.2 [48.3, 58.0]0.068
GPT-5.6 Terra54.051.8 [46.7, 57.0]0.002
Claude Opus 555.550.6 [45.8, 55.4]0.001
GLM-5.253.950.6 [45.7, 55.6]0.002
Claude Sonnet 551.647.6 [42.7, 52.9]<0.001
GPT-5.6 Luna49.046.0 [41.1, 50.9]<0.001
MiniMax M345.344.6 [39.7, 49.3]<0.001
DeepSeek V4 Flash43.340.6 [36.2, 45.2]<0.001
Mistral Medium 3.537.739.8 [35.3, 44.5]<0.001
GPT-5.4 Nano36.634.1 [29.4, 38.5]<0.001
GPT-5.4 Mini40.132.1 [26.9, 36.9]<0.001
Grok 4.20 Reasoning15.719.5 [15.5, 23.7]<0.001

The top model is significantly ahead of 11 of the 13 other models, and the ranking under the two metrics is nearly identical (Spearman ρ = 0.97). We agree that the research subset is small, and in the revision we present research results as a smaller-sample secondary analysis.

Regarding contamination, the engineering questions come from our own rollouts that were not public before the release, and all candidates are newly written descriptions, so these questions and answers did not exist in any corpus before the release. We will also release all 4,657 candidate forks with the generator's decisions, so that the selection can be audited.


W3: Frontier models are never evaluated without the prefix.

Thank you for this excellent suggestion. We evaluated six models across the ranking on all 502 questions with the trajectory prefix removed, keeping the questions, option order and scoring unchanged πŸ“[App. D.3]. Removing the trajectory lowers the accuracy of the top model from 58.2 to 53.4 (p = 0.02). The effect is clearest on the research questions, where long-horizon context matters most, and where removing the trajectory lowers the accuracy of five of the six models:

Research questionsWith trajectoryWithout trajectory
GPT-5.6 Sol62.557.1
GLM-5.259.851.8
GPT-5.6 Terra58.049.1
GPT-5.6 Luna54.540.2
MiniMax M346.442.9
DeepSeek V4 Flash48.252.7
Mean of 6 models54.949.0 (p = 0.009)

Across all question types, the mean of the six models is:

Subset (mean of 6 models)With trajectoryWithout trajectory
Research54.949.0
Parallel engineering66.861.3
Detour engineering37.538.2

Before these runs, we verified that our current API access reproduces the reported full-prefix results exactly on a 50-question check (GPT-5.6 Sol 62.0 vs 62.0, GLM-5.2 48.0 vs 48.0), so the differences above come from removing the trajectory.

In the revision, we report the no-prefix accuracy as a reference for every evaluated model πŸ“[L267]. Checking whether a model's score predicts its own rollouts is not possible with our forks, since each branch is taken only once by construction, and we list it as future work.

We hope these clarifications and the additional evidence address your concerns. Please let us know if you have any further questions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.