- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether an agent should follow the user instruction exactly, or treat it as a clue and figure out what the task really needs. A study of 100 real user-agent chats from SWE-chat finds that 56% of first instructions miss requirements that later turn out to be needed, and 52% of the useful follow-up requirements could have been worked out from the project itself. The authors call the missing skill outcome alignment and build TiGanEval to measure it. The benchmark has 64 software engineering and research tasks. Each task freezes five machine-checkable true requirements before the run starts, the first instruction states at most two of them, and every requirement can be discovered from the workspace. A simulated user gives short feedback about what went wrong after each failed submission, up to four submissions. A Full condition with the complete spec and a Literal condition that bans inference serve as controls. Tasks are written by a four-agent pipeline called TiGanTrans and then checked by humans. Tests on twelve models show that frontier models lose 53 to 67 points of Success@1 when moving from the full spec to the partial instruction. Four rounds of feedback do not close the gap, and a plan mode does not help either. An audit of 100 failed runs finds that 72% of failures come from using evidence badly rather than from bad execution.
Critical strengths
-
A clear problem statement with a clean experimental design. Splitting requirements into stated, inferred, and true ones turns a vague complaint about agents into something measurable. The Literal, Partial, and Full conditions share the same workspace, evaluator, and budget, so the gap between them isolates requirement inference from coding skill. The main result, a 53 to 67 point drop in Success@1 from Full to Partial that four rounds of feedback cannot close, is easy to read and useful for agent developers.
-
Careful measurement work, which is rare for this kind of benchmark. Requirements are frozen and machine-checkable, scoring uses held-out workloads, the synthesis pipeline has audit and calibration steps plus human review, repeated runs of two models stay within 4.3 Score points, and a resampling study finds 96% of simulated user messages faithful. The audit of 100 failed runs gives concrete failure modes, like the agent defending its own made-up rule against the project records, or ignoring a violation that its own replay printed. This tells the community what to fix.
Critical weaknesses
-
The Partial and Literal conditions differ in more than the instruction. The Partial prompt openly tells the agent to treat user messages as evidence and to check records and run experiments, while the Literal prompt bans inference. Part of the measured gap may be prompt following rather than a real ability difference. A neutral prompt condition would make the claim stronger.
-
The scale is small and the task pipeline may be biased. There are 64 tasks, most models run once per condition, every task has the same five-requirement shape, and all tasks come from one Claude Opus 5 pipeline with no reported human review statistics. The suite may reflect what one model family finds easy or hard to discover.
Questions for authors
- Can you add a neutral prompt that neither bans nor encourages inference, to separate the effect of the Partial prompt from the model's own behavior?
- Can you report human review statistics for TiGanTrans, such as how many tasks were rejected or repaired per criterion, and check whether models from the same family as the synthesis model find requirements easier to discover?
- On research tasks the mean Full@1 Success is only 49%. How do you tell the outcome cost apart from task or evaluator noise when the full spec already fails about half the time?
4 comments
W2, Q2. Scale, task shape, and possible pipeline bias
Human review statistics. TiGanTrans produced 109 candidates; 64 were retained, the 45 exclusions being made mainly for environment defects that could not be repaired and for departures from realistic practice. Each task was reviewed independently by two trained researchers, each of whom repaired it, and accepted only when both agreed. Over the review iterations of the 64 retained tasks, reviewers raised 64 flags on consistency among requirements, evaluator and observations, 46 on instruction plausibility, 55 on whether evidence suffices to infer each requirement, and 26 on leakage. Every retained task needed human repair, which shows where automated synthesis currently falls short; we will add these statistics to the appendix.
Task shape. The five equal-weight binary predicates standardize scoring only; the constraints and their discovery paths differ across tasks. SWE requirements range from ordering and concurrency invariants (e.g. concurrent map merges, bulk-import checkpoints) to parsing and validation rules (e.g. streaming JSON reading, conditional validation); research requirements govern estimation, uncertainty and stopping decisions (e.g. sequential rollout stopping, source apportionment). Discovery paths run through code, replayable records and experiments the agent writes. The SWE seeds come from DeepSWE, written independently of our pipeline.
Run-to-run variation. Beyond the three-run replications in Section 5.6, paired task-bootstrap CIs show that both the initial and the post-feedback gaps exclude zero for all eight frontier models (table in our response to Reviewer JSEE).
Same-family bias. If tasks generated by Claude Opus 5 favored what Claude models find discoverable, a Claude model should do comparatively well under Partial. We evaluated Claude-Sonnet-5 before stopping for cost reasons; on the 11 SWE tasks it completed under both conditions:
| Model | Partial Score@1 | Partial Success@1 | Partial Success@4 |
|---|---|---|---|
| Claude-Sonnet-5 | 47.3 | 18.2 | 54.5 |
| GPT-5.6-Sol | 60.0 | 9.1 | 18.2 |
| GPT-5.6-Terra | 61.8 | 0.0 | 54.5 |
| GPT-5.6-Luna | 50.9 | 0.0 | 18.2 |
| Kimi-K3 | 92.7 | 63.6 | 90.9 |
| MiMo-v2.6-Pro | 89.1 | 63.6 | 90.9 |
| DeepSeek-V4.1-Flash | 85.5 | 45.5 | 100.0 |
| GLM-5.3-Flash | 63.6 | 27.3 | 45.5 |
| MiMo-v2.6-Flash | 81.8 | 54.5 | 81.8 |
Sonnet has the lowest Partial Score@1 of the nine models although it runs the most experiments (43.2 per trajectory vs. 3.9-15.9), and its Full@1 Success on the same tasks is 81.8%, a 63.6-point gap. We find no same-family advantage, though 11 tasks limit the strength of this check. On the full suite, the gap is also not specific to any family: models from five families lose 53.1-67.2 Success@1 points.
Q3. Separating the research outcome cost from noise
A Full@1 Success of about 49% could reflect three things other than the cost of the missing specification: evaluator noise, tasks that are hard regardless of the instruction, and run-to-run randomness. We address each in turn.
- Evaluator noise: Full failures are substantive scientific errors. Requirements are fixed programmatic checks on the saved delivery, not an LLM judge. Of the 98 failed Full research trajectories, 91 fail at least one uncertainty-estimation or experiment-design check. For example, GPT-5.6-Sol clipped a computed uncertainty interval to the requested precision and used the clipped interval to stop collecting data; re-evaluating the saved artifact reproduces the failure, and removing only the clipping raises the Score from 3/5 to 4/5.
- Task difficulty: the cost is measured within each task. The outcome cost is the Full − Partial difference for the same model on the same task, scored by the same checks. Difficulty that a task has regardless of the instruction lowers both conditions and does not enter this difference; if anything, a task that already fails under Full leaves Partial less room to fall, so the measured cost is conservative. Resampling the 24 research tasks (50,000 resamples, differences computed per model and task and averaged over the eight models) shows that the difference is clearly above zero:
| Comparison | Score gap [95% CI] | Success gap [95% CI] |
|---|---|---|
| Full@1 − Partial@1 | 43.1 [36.0, 50.3] | 45.8 [32.3, 59.4] |
| Full@1 − Partial@4 | 35.7 [28.6, 42.9] | 37.5 [24.0, 51.0] |
- Run-to-run randomness: the cost persists where Full succeeds reliably. For GPT-5.6-Luna and DeepSeek-V4.1-Flash, eight model-task pairs succeed under Full in all three runs, so on these pairs neither the task nor chance prevents success. Under Partial, the same pairs succeed in 1 of 24 runs at the first submission and in 4 of 24 after four submissions.
We will add this analysis to Section 5.2 and Appendix I.
I thank the authors for the thorough rebuttal with new experiments.
[W1, Q1]: Thanks for clarifying that Full and Partial share the same prompt, and the new Neutral condition nicely shows the gap comes from the models. [W2, Q2]: The human review statistics are exactly what I asked for, and the Claude-Sonnet-5 check is a clean test of same-family bias. [Q3]: The three-part answer is solid.
Overall, all three concerns are well addressed. I will maintain my score.
Thank you for your feedback. If you have any further questions, please feel free to let us know at any time.
Thank you for the review. You raise three concerns: a prompt confound between Partial and Literal, the scale and possible bias of the pipeline, and whether the research outcome cost can be separated from noise when Full succeeds only about half the time. In short:
W1, Q1. Prompt confound and a neutral condition
The main measurement compares Full and Partial, which share the same system prompt and differ only in the initial instruction and the owner's knowledge (Appendix F.1), so it carries no prompt confound. The Partial-Literal contrast is a prompt manipulation by design; it estimates how much inference contributes, and we will present Literal as a reference rather than as part of the main comparison.
To separate the Partial prompt from the model's own behavior, we added a Neutral condition on all 64 tasks for three models: "You are a helpful assistant that can interact with a computer. Work in the project workspace through the available tools." It neither encourages nor prohibits inference; the instruction, workspace, tools, protocol and four-submission budget are those of Partial.
Removing the investigation guidance lowers first-submission success for Kimi-K3 and DeepSeek-V4.1-Flash and leaves GPT-5.6-Luna unchanged. Without it, models do not infer more on their own: Neutral Success@1 is 3.1-18.8% against 60.9-81.3% under Full. The low Partial performance reflects the models, not the prompt.