- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper defines "taste" as the ability of LLM to choose the solution path that leads to the better outcome in long horizon. It builds TASTE-BENCH, 502 binary-choice questioned automatically mined from trajectoreies on SWE-bench, using parallel or detour trajectories. Then the paper shows a number of analyses on the challengingness of the dataset, and how models behave, and some distillation results on how we can use the dataset to further improve models.
Critical strengths
- carefully designed annotation-free construction of a dataset
- transparent evaluations
Critical weaknesses
- I don't see how this is different from RL. Picking the better option at a form is basically estimating which action has the higher Q-value, and the labels come from single rollouts, which is what RL methods and process reward models already do. The paper doesn't explain what this captures that RL can't, and there's no comparison with RL, PRM, or step-level DPO trained on the same data. (This is the most severe issue of this paper)
- The labels may not really reflect the decision at the fork, because the two attempts don't actually share the "identical" prefix in parallel forks (it says it follows success trajectories' ones), and good option was only tried after the agent saw the failure in detour forks. This should have been made clear and double checked empirically whether this doesn't make much implications. While human judgements were used to assess the quality of the dataset, I do not agree with the arguments, as the human reviewers have observed the outcome. So it is merely a sanity check whether there were no systematic errors that checks wrong outputs as the right output.
- In experiment figure 3, while it is argued that the state of the art models have poor taste of 59.7%, we do not know whether it is because of poor taste or it is because of bad dataset that no one can guess. Upper bound results should have been provided, e.g., human accuracy. The question generator is also the same as the top-ranked model, so the results may also have been biased.
- The end-to-end experiment in distillation experiment does not isloate student's own contribution. The advice is written from earlier runs on the same task with knowledge of the hidden test results, so even wrong advice may give the agent useful hints. a few more baselines needed to clarify.
Questions for authors
- How is this different from training a process reward model or doing RL on the same rollouts? Could you add a baselines like PRM or step-level DPO? Why can't 'taste' learnt from usual RL training simply from SWE-bench?
- What is the human accuracy before seeing the outcome?
- Does the ranking change if you use a different model as the generator?
- In the end-to-end experiment, what happens with random advice, or with both options shown without a recommendation?
3 comments
Response to Reviewer M3Xv (2/3)
W2: The labels may not reflect the decision at the fork, and the human reviewers observed the outcome.
Thank you for pointing this out. For detour forks, the generator rejects any fork whose better direction can only be named after seeing the failure, and it checks that the prefix does not reveal the failure or the later fix π[L150]. For parallel forks, the two attempts share the same task and diverge at the fork, and a question is released only when all four judges confirm the label from the full record. Regarding the human review, we agree that judging after seeing the outcome only checks the labels. We therefore also report the human accuracy before the outcome is shown, which we describe in our next response (W3 / Q2).
To test whether the label depends on which attempt provides the history, we also re-evaluated all 172 parallel questions with the prefix taken from the failing attempt instead of the passing one:
| Model | History from the passing attempt (paper) | History from the failing attempt | Questions with the same answer |
|---|---|---|---|
| GPT-5.6 Sol | 70.3 | 58.7 | 79% |
| GLM-5.2 | 63.4 | 55.2 | 74% |
With either history, both models remain well above the 25% of random guessing, and most answers are unchanged π[App. D.5].
W3 / Q2 / Q3: Is 59.7% low because of poor taste or because the dataset is unanswerable? What is the human accuracy before seeing the outcome? Does the ranking change with a different generator?
Thank you for this important question. We understand your concern to be: without a reference point, a low score could reflect noisy labels rather than limited taste.
We first note that our headline metric counts a question as correct only when the model chooses the better direction in both candidate orders. Random guessing therefore scores 25% rather than 50%, and a model that always prefers the same position scores 0%. On this scale, 59.7% is well above chance, and the remaining question is how far it is from what a human can achieve.
Our two-stage human review helps to answer this question. Each reviewer first chooses a direction before any continuation or outcome is shown, and only then sees the recorded outcomes and judges which decision was better. The second stage measures label quality, and the first stage gives a human reference at decision time.
| Label quality (after seeing the outcome) | Result |
|---|---|
| Human judgments that support the mined label | 98.8% (170 of 172) |
| Agreement between the two reviewers | Cohen's ΞΊ = 0.97 |
| Accuracy at decision time (68 shared questions) | Accuracy |
|---|---|
| Human reviewer 1 | 75.0 |
| Human reviewer 2 | 57.4 |
| Best model (GPT-5.5) | 66.2 |
| Median of 14 models | 55.9 |
The mined labels are almost always confirmed once the outcome is known, and the more experienced reviewer outperforms every evaluated model before the outcome is known. Together, these results indicate that the labels are reliable and that the 59.7% of the best model reflects the difficulty of the decisions rather than noise in the dataset. We have added this human reference to the revision π[L265βL267], π[App. F.2].
Regarding the generator, GPT-5.6 Sol is statistically tied with GPT-5.5 on Taste-Bench, the two smaller models of the same family rank 5th and 8th, and the generator is excluded from the judge panel that filters the questions. Re-building the benchmark with another generator is beyond the rebuttal period, and we will add a generator-swap study on a subset in the final version.
Response to Reviewer M3Xv (3/3)
W4 / Q4: The end-to-end experiment does not isolate the student's contribution. What happens with random advice, or with both options shown without a recommendation?
Thank you for this insightful point. We agree that the notes themselves carry information about the task. We understand your concern to be: the end-to-end gain could come from the task information in the notes rather than from the judgment of the student.
The contribution of the student is its judgment at each decision point. To isolate it, we compared the student with the untrained base model given exactly the same fork questions, prompt and decision rule on the 98 held-out forks:
| Advisor | Decision points where the advisor picks the better direction |
|---|---|
| Untrained base model | 55.1% |
| Distilled student | 78.6% (p = 0.0015, 18 tasks better and 3 worse) |
Since both advisors receive the same historical information, this comparison separates the distilled judgment from the information in the notes. The student is never trained on these tasks, because training and evaluation use disjoint sets of tasks.
We also ran the controls you suggested. On the paper's executor, we compare different advisors on the same 41 held-out tasks:
| Advice given to the executor agent | Task success |
|---|---|
| No advice | 14.6% |
| Untrained base model | 30.8% |
| Random advice | 34.1% |
| Distilled student | 33.7% |
| Correct advice at every fork | 39.0% |
In addition, we ran notes that show both options without a recommendation on a separately deployed copy of the same executor agent. Since this deployment differs from the paper, we compare each type of notes only with the run without notes on the same deployment:
| Notes given to the executor agent | Change in task success over no notes (percentage points) |
|---|---|
| Note header and submission checklist only, without decision points | +0.0 |
| Decision points with both options, without a recommendation | +10.5 |
| Decision points recommending the worse option | +10.5 |
| Decision points recommending the better option | +15.8 |
These controls show that the decision points in the notes carry useful information about the task, which is why we use the fork-level comparison above to isolate the judgment of the student.
In the original manuscript, we present the end-to-end success rate as the main evidence for the student. In the revision, we use this fork-level accuracy as the primary measure of the student's contribution, present the end-to-end results as supporting evidence, and revise Finding 6 accordingly π[L442βL443], π[App. H].
We hope these clarifications and the additional evidence address your concerns. Please let us know if you have any further questions!
Response to Reviewer M3Xv (1/3)
We thank the reviewer for the thoughtful and constructive feedback. We are encouraged that the reviewer finds our dataset construction carefully designed and annotation-free, and our evaluations transparent. Following your questions, we conducted several new analyses and revised the paper accordingly. We carefully address your concerns point by point below.
Thank you for raising this central question. We understand your concern to be: if choosing the better option at a fork is equivalent to estimating which action has the higher value, what does Taste-Bench capture that RL, PRMs or step-level DPO do not?
We would like to clarify that Taste-Bench is a benchmark rather than a training method. PRMs, value functions and RL policies are possible ways to make the decisions in our questions, and Taste-Bench provides outcome-labeled test items, with the full decision-time context, on which any of these estimators can be evaluated. In this sense, our work is complementary to RL and PRMs rather than an alternative to them.
To examine whether the benchmark can be solved by evaluating each step in isolation, we compared our standard protocol, where the model sees both candidates and chooses one, with pointwise scoring in the style of a PRM, where the model sees one candidate at a time and outputs the probability that the agent succeeds after taking it (150-question subset).
Pointwise scoring does not improve accuracy for any of the three models, and it often assigns the same score to both candidates. We have added this comparison to the revision π[Sec. 4.2, L267βL268], π[App. D.4].
In the original manuscript, two further results show that Taste-Bench measures long-horizon judgment rather than the evaluation of a single step:
Accuracy falls monotonically as the deciding evidence appears further in the future, close to the 25% of random guessing at the longest horizon (Section 4.3). In addition, a larger reasoning budget does not help: a 17-fold increase in reasoning tokens changes the accuracy of GPT-5.6 Sol by only β0.2 points (Section 4.4). These findings suggest that the questions require anticipating the consequences of a direction, which a model cannot obtain by evaluating the current step more carefully.
Regarding RL on SWE-bench, an episode-level reward does not identify which of the many decisions in a long trajectory caused the failure. Our mining step turns such trajectories into decision-level supervision, two candidates at one state with an outcome label, and these items can serve as training and evaluation data for RL and PRM methods in future work.
To verify this, we also ran a pilot that trains the same base model directly on the binary fork label, using 215 parallel engineering questions from an earlier version of the question pool and two task-disjoint folds:
Neither method improves held-out accuracy over the base model. This pilot motivates our choice to distill the reasoning of a teacher model that knows the correct answer, instead of fitting the label π[App. G].