acceptodds
Under review as a conference paper at ICLR 2027

Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents

Abstract

Large language model agents can follow complex human instructions to solve real-world tasks. In practice, however, initial instructions are often incomplete. In our analysis of 100 user-agent trajectories from SWE-chat, 56% of initial instructions omit requirements that later emerge as necessary, and 52% of the useful follow-up requirements could have been inferred from the project. A user states requirements from their own understanding of the task, and these **stated requirements** often differ from the **true requirements** that completing the task imposes. An agent should therefore treat the instruction as evidence rather than as the objective, inferring the true requirements from the instruction and the environment and satisfying them even when the user is not yet aware of all of them. We call this ability **outcome alignment**. In this work, we construct **TiGanEval**, a benchmark of software engineering and research tasks. In every task the user genuinely lacks part of the requirements, each requirement stays discoverable from the code and runtime behaviour of the environment, and requirements the agent infers are separated from those the user adds after observing delivered work. These properties together make outcome alignment measurable. Evaluating eight frontier models including GPT-5.6-Sol, we find outcome alignment broadly missing: the models lose 53 to 67 points of success rate when moving from the full specification to the partial instruction, and consequence feedback across four submissions does not close the gap. Furthermore, a plan-mode harness cannot compensate for the missing ability of the model itself, which places outcome alignment among the abilities the next generation of models should be developed for.

Primary area:
datasets and benchmarks
Keywords:
LLM agent evaluation, outcome alignment, autonomous agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pdmAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether an agent should follow the user instruction exactly, or treat it as a clue and figure out what the task really needs. A study of 100 real user-agent chats from SWE-chat finds that 56% of first instructions miss requirements that later turn out to be needed, and 52% of the useful follow-up requirements could have been worked out from the project itself. The authors call the missing skill outcome alignment and build TiGanEval to measure it. The benchmark has 64 software engineering and research tasks. Each task freezes five machine-checkable true requirements before the run starts, the first instruction states at most two of them, and every requirement can be discovered from the workspace. A simulated user gives short feedback about what went wrong after each failed submission, up to four submissions. A Full condition with the complete spec and a Literal condition that bans inference serve as controls. Tasks are written by a four-agent pipeline called TiGanTrans and then checked by humans. Tests on twelve models show that frontier models lose 53 to 67 points of Success@1 when moving from the full spec to the partial instruction. Four rounds of feedback do not close the gap, and a plan mode does not help either. An audit of 100 failed runs finds that 72% of failures come from using evidence badly rather than from bad execution.

Critical strengths
  1. A clear problem statement with a clean experimental design. Splitting requirements into stated, inferred, and true ones turns a vague complaint about agents into something measurable. The Literal, Partial, and Full conditions share the same workspace, evaluator, and budget, so the gap between them isolates requirement inference from coding skill. The main result, a 53 to 67 point drop in Success@1 from Full to Partial that four rounds of feedback cannot close, is easy to read and useful for agent developers.

  2. Careful measurement work, which is rare for this kind of benchmark. Requirements are frozen and machine-checkable, scoring uses held-out workloads, the synthesis pipeline has audit and calibration steps plus human review, repeated runs of two models stay within 4.3 Score points, and a resampling study finds 96% of simulated user messages faithful. The audit of 100 failed runs gives concrete failure modes, like the agent defending its own made-up rule against the project records, or ignoring a violation that its own replay printed. This tells the community what to fix.

Critical weaknesses
  1. The Partial and Literal conditions differ in more than the instruction. The Partial prompt openly tells the agent to treat user messages as evidence and to check records and run experiments, while the Literal prompt bans inference. Part of the measured gap may be prompt following rather than a real ability difference. A neutral prompt condition would make the claim stronger.

  2. The scale is small and the task pipeline may be biased. There are 64 tasks, most models run once per condition, every task has the same five-requirement shape, and all tasks come from one Claude Opus 5 pipeline with no reported human review statistics. The suite may reflect what one model family finds easy or hard to discover.

Questions for authors
  1. Can you add a neutral prompt that neither bans nor encourages inference, to separate the effect of the Partial prompt from the model's own behavior?
  2. Can you report human review statistics for TiGanTrans, such as how many tasks were rejected or repaired per criterion, and check whether models from the same family as the synthesis model find requirements easier to discover?
  3. On research tasks the mean Full@1 Success is only 49%. How do you tell the outcome cost apart from task or evaluator noise when the full spec already fails about half the time?
4 comments

Thank you for the review. You raise three concerns: a prompt confound between Partial and Literal, the scale and possible bias of the pipeline, and whether the research outcome cost can be separated from noise when Full succeeds only about half the time. In short:

  • Prompt confound: the 53-67-point outcome cost compares Full and Partial, which use the identical prompt. Under a new Neutral prompt, first-submission success stays far below Full.
  • Pipeline: we report human review statistics; a Claude model shows no same-family advantage, and the gap holds for every model family.
  • Research noise: the checks are fixed programs, Full failures are substantive scientific errors, and where a model succeeds under Full in all three runs, Partial@1 succeeds in 1 of 24 runs.

W1, Q1. Prompt confound and a neutral condition

The main measurement compares Full and Partial, which share the same system prompt and differ only in the initial instruction and the owner's knowledge (Appendix F.1), so it carries no prompt confound. The Partial-Literal contrast is a prompt manipulation by design; it estimates how much inference contributes, and we will present Literal as a reference rather than as part of the main comparison.

To separate the Partial prompt from the model's own behavior, we added a Neutral condition on all 64 tasks for three models: "You are a helpful assistant that can interact with a computer. Work in the project workspace through the available tools." It neither encourages nor prohibits inference; the instruction, workspace, tools, protocol and four-submission budget are those of Partial.

ModelScore@1: Partial / NeutralSuccess@1: Partial / NeutralSuccess@1: Full
GPT-5.6-Luna41.6 / 41.93.1 / 3.160.9
Kimi-K368.8 / 60.021.9 / 14.175.0
DeepSeek-V4.1-Flash66.3 / 64.425.0 / 18.881.3

Removing the investigation guidance lowers first-submission success for Kimi-K3 and DeepSeek-V4.1-Flash and leaves GPT-5.6-Luna unchanged. Without it, models do not infer more on their own: Neutral Success@1 is 3.1-18.8% against 60.9-81.3% under Full. The low Partial performance reflects the models, not the prompt.

W2, Q2. Scale, task shape, and possible pipeline bias

Human review statistics. TiGanTrans produced 109 candidates; 64 were retained, the 45 exclusions being made mainly for environment defects that could not be repaired and for departures from realistic practice. Each task was reviewed independently by two trained researchers, each of whom repaired it, and accepted only when both agreed. Over the review iterations of the 64 retained tasks, reviewers raised 64 flags on consistency among requirements, evaluator and observations, 46 on instruction plausibility, 55 on whether evidence suffices to infer each requirement, and 26 on leakage. Every retained task needed human repair, which shows where automated synthesis currently falls short; we will add these statistics to the appendix.

Task shape. The five equal-weight binary predicates standardize scoring only; the constraints and their discovery paths differ across tasks. SWE requirements range from ordering and concurrency invariants (e.g. concurrent map merges, bulk-import checkpoints) to parsing and validation rules (e.g. streaming JSON reading, conditional validation); research requirements govern estimation, uncertainty and stopping decisions (e.g. sequential rollout stopping, source apportionment). Discovery paths run through code, replayable records and experiments the agent writes. The SWE seeds come from DeepSWE, written independently of our pipeline.

Run-to-run variation. Beyond the three-run replications in Section 5.6, paired task-bootstrap CIs show that both the initial and the post-feedback gaps exclude zero for all eight frontier models (table in our response to Reviewer JSEE).

Same-family bias. If tasks generated by Claude Opus 5 favored what Claude models find discoverable, a Claude model should do comparatively well under Partial. We evaluated Claude-Sonnet-5 before stopping for cost reasons; on the 11 SWE tasks it completed under both conditions:

ModelPartial Score@1Partial Success@1Partial Success@4
Claude-Sonnet-547.318.254.5
GPT-5.6-Sol60.09.118.2
GPT-5.6-Terra61.80.054.5
GPT-5.6-Luna50.90.018.2
Kimi-K392.763.690.9
MiMo-v2.6-Pro89.163.690.9
DeepSeek-V4.1-Flash85.545.5100.0
GLM-5.3-Flash63.627.345.5
MiMo-v2.6-Flash81.854.581.8

Sonnet has the lowest Partial Score@1 of the nine models although it runs the most experiments (43.2 per trajectory vs. 3.9-15.9), and its Full@1 Success on the same tasks is 81.8%, a 63.6-point gap. We find no same-family advantage, though 11 tasks limit the strength of this check. On the full suite, the gap is also not specific to any family: models from five families lose 53.1-67.2 Success@1 points.

Q3. Separating the research outcome cost from noise

A Full@1 Success of about 49% could reflect three things other than the cost of the missing specification: evaluator noise, tasks that are hard regardless of the instruction, and run-to-run randomness. We address each in turn.

  1. Evaluator noise: Full failures are substantive scientific errors. Requirements are fixed programmatic checks on the saved delivery, not an LLM judge. Of the 98 failed Full research trajectories, 91 fail at least one uncertainty-estimation or experiment-design check. For example, GPT-5.6-Sol clipped a computed uncertainty interval to the requested precision and used the clipped interval to stop collecting data; re-evaluating the saved artifact reproduces the failure, and removing only the clipping raises the Score from 3/5 to 4/5.
  2. Task difficulty: the cost is measured within each task. The outcome cost is the Full − Partial difference for the same model on the same task, scored by the same checks. Difficulty that a task has regardless of the instruction lowers both conditions and does not enter this difference; if anything, a task that already fails under Full leaves Partial less room to fall, so the measured cost is conservative. Resampling the 24 research tasks (50,000 resamples, differences computed per model and task and averaged over the eight models) shows that the difference is clearly above zero:
ComparisonScore gap [95% CI]Success gap [95% CI]
Full@1 − Partial@143.1 [36.0, 50.3]45.8 [32.3, 59.4]
Full@1 − Partial@435.7 [28.6, 42.9]37.5 [24.0, 51.0]
  1. Run-to-run randomness: the cost persists where Full succeeds reliably. For GPT-5.6-Luna and DeepSeek-V4.1-Flash, eight model-task pairs succeed under Full in all three runs, so on these pairs neither the task nor chance prevents success. Under Partial, the same pairs succeed in 1 of 24 runs at the first submission and in 4 of 24 after four submissions.

We will add this analysis to Section 5.2 and Appendix I.

I thank the authors for the thorough rebuttal with new experiments.

[W1, Q1]: Thanks for clarifying that Full and Partial share the same prompt, and the new Neutral condition nicely shows the gap comes from the models. [W2, Q2]: The human review statistics are exactly what I asked for, and the Claude-Sonnet-5 check is a clean test of same-family bias. [Q3]: The three-part answer is solid.

Overall, all three concerns are well addressed. I will maintain my score.

Thank you for your feedback. If you have any further questions, please feel free to let us know at any time.

Reviewer_JSEEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces outcome alignment, where agents infer the true task requirements beyond incomplete or mistaken user instructions. It proposes TiGanEval and the TiGanTrans pipeline to evaluate this ability with verifiable hidden requirements and matched partial/full specifications. Experiments show a large gap between full and partial specifications, and neither feedback nor simple planning fully closes it.

Critical strengths

• Important and well-motivated problem.The paper identifies a practically meaningful limitation of current instruction-following agents: successful task completion may require requirements that users themselves have not explicitly stated. The SWE-chat analysis provides useful empirical motivation for this setting. • Clean and well-controlled benchmark design.The matched Partial/Full/Literal settings make it possible to isolate the effect of missing requirements and autonomous requirement inference, while keeping the workspace, evaluator, tools, and task largely fixed. • Strong diagnostic analysis beyond aggregate scores.The paper analyzes when requirements are resolved and audits failure trajectories, showing that many failures arise from poor use of available evidence rather than execution alone.

Critical weaknesses

• Construct validity of “true requirements” is unclear.The “true requirements” are not independently grounded in real user outcomes; instead, they are distilled from fully specified source tasks by the benchmark construction pipeline and then deliberately hidden from the agent. Therefore, TiGanEval may primarily measure an agent’s ability to recover designer-specified latent requirements from environmental evidence, rather than the broader notion of outcome alignment with genuinely underspecified real-world user needs. • The benchmark may use unrealistically severe underspecification.Every task has five true requirements, yet 39% of initial instructions cover none and 50% cover only one. While the motivating SWE-chat study shows that incomplete instructions are common, it does not establish that real-world specification gaps are typically this severe. Therefore, the reported 53–67 point performance drop may partly reflect the deliberately large gaps constructed in TiGanEval rather than the magnitude of outcome-alignment failures encountered in realistic interactions. • Some broad conclusions are supported by relatively limited evaluation.The benchmark contains only 64 tasks, and the claim that harness engineering cannot compensate for outcome-alignment deficiencies is based on one particular plan-mode design evaluated on only three models. • The novelty relative to prior work is not sufficiently established.Prior work such as Implicit Intelligence [1] and SWE-RPG [2] already studies agents recovering unstated or implicit requirements from the environment. TiGanEval adds a more controlled protocol, but the paper should clarify what fundamentally new capability it measures beyond existing implicit requirement discovery benchmarks. [1] Implicit Intelligence — Evaluating Agents on What Users Don’t Say. ICML 2026.Proceedings of Machine Learning Research [2] A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents (SWE-RPG). 2026.

Questions for authors

• How representative is the severity of TiGanEval's specification gaps of real user-agent interactions?Could the authors compare the fraction of initially stated requirements in TiGanEval with the SWE-chat trajectories and report performance stratified by gap size? • How should “true requirements” be interpreted when multiple reasonable solutions exist?Could different annotators define different valid requirement sets for the same project, and if so, how does the benchmark distinguish alternative valid solutions from failures of outcome alignment? • How robust are the conclusions to stronger agent scaffolds?Have the authors considered mechanisms explicitly designed for requirement discovery, such as hypothesis tracking, mandatory evidence collection, or iterative requirement verification, rather than only a generic plan-then-execute harness?

6 comments

Thank you for the review. You raise four concerns: what grounds the "true requirements", whether the gaps are unrealistically severe, whether 64 tasks and one plan-mode design support the conclusions, and what TiGanEval measures beyond implicit-requirement benchmarks. In short:

  • True requirements are the requirements a user demands once they observe what violating them does; every one is fixed in advance, determined by project evidence, and attached to such an observation.
  • Severity: the 53-67-point drops are gaps on TiGanEval's tasks, not real-world averages, and we will say so; the gap remains above 56 Success points even when two of five requirements are stated.
  • Scale and harness: task-bootstrap CIs exclude zero for all models; a harness built from the mechanisms you list does not change Success@1.
  • Novelty: in TiGanEval the missing requirement is outside the user's knowledge and the user's own proposal can be wrong, so neither asking nor faithful completion recovers it; the protocol measures whether the agent closes the gap before or only after the user observes a failure.

W1, Q2. What grounds the "true requirements"?

Each requirement is defined by a user-visible outcome. Every requirement in R comes with an authored observation: the consequence a user encounters when it is violated, such as an order charged twice. When the simulated user sees that observation, it asks for the behavior to change (96% of resampled messages convey exactly the observed information, Section 5.6). So each true requirement is one the user would demand once they saw its violation; at the start, the user lacks the observation, not the preference. This is the structure we found in SWE-chat, where 52% of useful follow-up requirements could have been inferred from the project before the user stated them (Appendix B). For example, a user who asked for a branch review later asked whether an agent integration kept complete histories when resuming sessions; the export path relied on messages the running plugin had observed, so earlier messages could be lost, although the project documentation already required complete transcripts. TiGanEval fixes such requirements before the episode, which makes them verifiable.

Three further properties keep R from reflecting a designer's preference:

  1. Origin. R is distilled from a fully specified source task; for SWE these are DeepSWE tasks written independently of our pipeline.
  2. Determined by evidence, checked by two reviewers. Review criterion (3) requires that workspace evidence suffice to infer each requirement and to determine the required behavior. Two trained researchers review each task independently, and it is accepted only when both agree; reviewers raised 55 flags on this criterion across the 64 tasks. This answers Q2: annotators may phrase a requirement differently, but a requirement that the same evidence would justify in incompatible ways does not pass review.
  3. Outcome predicates, not reference implementations. Each requirement checks delivered behavior on held-out workloads, so any implementation with that behavior passes. Under Full, the same models satisfy all five requirements on 60.9-82.8% of tasks, so the requirement sets are coherent and achievable; the Partial drop isolates the inference the instruction leaves to the agent.

The scope is requirements that the project determines. Preferences that exist only in a user's mind cannot be recovered from the environment and are the subject of clarification benchmarks; we will state this boundary in Section 3 and the Limitations.

W2, Q1. Are the specification gaps unrealistically severe?

The 53-67-point drops measure the gap on TiGanEval's tasks, not the average loss in real interactions; we will state this in the Abstract and Section 5.2. Three points bear on representativeness:

  • Coverage counts complete criteria. A requirement counts as covered only if the full criterion is stated or entailed. Another 117 of the 320 requirements are partially stated, so "0/5 covered" does not mean the instruction carries no information.
  • Relation to SWE-chat. TiGanEval corresponds to the 56% of SWE-chat sessions whose initial instruction is incomplete, not to all sessions. A closer comparison is limited because SWE-chat sessions have no fixed requirement set: later requests can introduce new goals.
  • Stratification. The gap persists at every coverage level (SWE tasks, mean over the eight frontier models, paired task-bootstrap 95% CIs):
Requirements fully covered initiallyTasksScore@1 gapSuccess@1 gap
0/51932.1 [25.7, 38.8]71.7 [61.8, 80.9]
1/51526.2 [18.5, 34.8]63.3 [48.3, 77.5]
2/5625.4 [12.5, 37.1]56.3 [31.3, 79.2]

The gap shrinks as coverage grows, as expected, but stays above 56 Success points with two of five requirements stated. Strata contain different tasks, so we do not attribute the differences to gap size alone.

W3, Q3. Scale and the harness conclusion

Scale. The 64 tasks were retained from 109 candidates, each reviewed and repaired by two researchers. Paired task-bootstrap CIs (50,000 resamples, stratified by domain), in percentage points:

ModelScore gap @1Success gap @1Score gap after 4 submissionsSuccess gap after 4 submissions
GPT-5.6-Sol40.0 [33.4, 46.6]53.1 [42.2, 64.1]26.9 [21.2, 32.8]34.4 [21.9, 46.9]
GPT-5.6-Terra41.2 [35.6, 47.2]64.1 [53.1, 75.0]27.2 [21.2, 33.1]40.6 [26.6, 53.1]
GPT-5.6-Luna42.5 [34.4, 50.3]57.8 [46.9, 68.8]31.2 [22.8, 39.1]37.5 [25.0, 50.0]
Kimi-K323.4 [16.9, 30.0]53.1 [40.6, 65.6]7.5 [2.2, 12.5]15.6 [3.1, 28.1]
MiMo-v2.6-Pro24.7 [17.5, 31.3]53.1 [39.1, 65.6]10.0 [3.8, 15.9]18.8 [4.7, 32.8]
DeepSeek-V4.1-Flash28.1 [20.9, 35.0]56.2 [43.8, 68.8]14.4 [7.5, 20.6]20.3 [9.4, 31.2]
GLM-5.3-Flash34.7 [28.8, 40.6]67.2 [56.2, 78.1]23.8 [18.1, 29.7]46.9 [34.4, 59.4]
MiMo-v2.6-Flash39.1 [30.9, 47.2]64.1 [51.6, 75.0]26.6 [17.8, 35.6]37.5 [25.0, 50.0]

Every interval excludes zero, including after four submissions.

Harness. Your Q3 names hypothesis tracking, mandatory evidence collection and iterative requirement verification. We built a harness with all three: it records requirement hypotheses, requires each to be linked to project evidence, runs checks that discriminate between competing interpretations, and revises the plan when new evidence contradicts it. On all 64 tasks, against Partial with the same model and a one-submission budget:

ModelScore@1Δ [95% CI]Success@1Δ [95% CI]
GPT-5.6-Luna40.3 → 42.5+2.2 [−3.4, 7.5]0.0 → 3.1+3.1 [0.0, 7.8]
Kimi-K362.5 → 62.50.0 [−5.6, 5.6]18.8 → 17.2−1.6 [−10.9, 7.8]
DeepSeek-V4.1-Flash65.6 → 70.9+5.3 [0.9, 9.7]26.6 → 26.60.0 [−9.4, 9.4]

DeepSeek-V4.1-Flash gains 13.3 [4.2, 23.3] Score points on research tasks, so TiGanEval registers harness improvements when they occur; no Success@1 change has an interval excluding zero. We will restrict the claim to the two designs tested.

W4. What does TiGanEval measure beyond implicit-requirement benchmarks?

We agree that recovering unstated requirements from the environment is shared ground. TiGanEval differs in why the requirement is unstated and in what the protocol measures.

Why the requirement is unstated. In Implicit Intelligence, the missing requirements are shared expectations the user holds but does not say (accessibility, privacy, irreversible actions). In SWE-RPG, the original issue states the goal and its implicit requirements are mined from the issue and PR discussion, the developer patch and the tests. In both, the request is correct but incomplete, so the right behavior is to complete it. In TiGanEval the user's understanding of the task is partial and may be wrong: the user proposes a mechanism, such as retrying a timed-out payment, that the project records contradict. The agent must sometimes revise the instruction rather than complete it, and asking cannot substitute, because the owner can confirm only what they know. Our failure audit shows this is a distinct failure: in 10% of failed episodes the agent implements the owner's mistaken belief over contrary records (Section 5.5), a failure that cannot arise when the request is correct.

What the protocol measures. Both prior benchmarks score one end-to-end attempt: Implicit Intelligence with an LLM judge over an LLM-simulated world, SWE-RPG with executable tests plus a retrospective LLM diagnosis of which stage failed. Neither has a user who can report consequences, and neither has a condition in which the full requirements are given. TiGanEval keeps both routes to the requirements open, the agent's own inference and the user's reports of consequences, and records which route resolves each requirement; the matched Full condition turns the comparison into a paired estimate of what the missing specification costs. This is what yields findings outside the scope of the prior settings: after the first submission, frontier models resolve at most 17.6% of the remaining unstated requirements before they are reported, and 40% of reported requirements are fixed in neither of the next two submissions (Table 5). Reviewer qv3Z also notes that this separation has not appeared in earlier interactive benchmarks.

Implicit IntelligenceSWE-RPGTiGanEval
Why the requirement is unstatedExpected by the user, not saidOmitted from a correct issueOutside the user's knowledge, or contradicted by the user's belief
Can the user's own proposal be wrong?NoNoYes
Does a user report consequences?NoNoYes, up to four submissions
Matched full-specification controlNoNoYes
VerificationLLM judge over an LLM-simulated worldExecutable testsFrozen executable checks

We will add Implicit Intelligence to Table 1 and Section 2 with this distinction, and acknowledge SWE-RPG's finding that requirement recovery is the main failure stage in natural issues, which our results complement.

Reviewer_qv3ZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes outcome alignment: treating a user's instruction as evidence of the underlying task objective rather than as a complete specification. TiGanEval evaluates this idea through 64 tasks, split into 40 SWE and 24 research tasks. Each contains five hidden, verifiable requirements, a simulated user with incomplete knowledge, and written feedback after unsuccessful submissions. An LLM pipeline, TiGanTrans, generates the tasks before human review. The authors report a 53 to 67-point reduction in frontier models' Success@1 when instructions are partial rather than complete. They further argue that neither four feedback rounds nor a plan mode eliminates the gap.

Critical strengths
  1. The paired comparison is a strength. Full and Partial share the workspace, tools and evaluator, with requirements fixed before execution, enabling task-level measurement of their performance difference. Table 5's separation of requirements satisfied before feedback from those corrected only afterward is particularly informative, and I have not seen that analysis in earlier interactive benchmarks.

  2. The failure analysis identifies behavior worth investigating: agents repair reported symptoms without recovering the underlying rule and often overlook violations already exposed by their own replays (Figure 5, Appendix J and K). These observations suggest concrete directions for improvement beyond what clarification benchmarks typically examine.

Critical weaknesses
  1. The main performance gap may combine the intended ability with differences in strategy and task construction. Agents know they have one submission under Full but up to four under Partial (Section 5.1, Appendix F.1), so early submission to obtain feedback is a plausible strategy under Partial. An equal-submission-budget control would test this explanation. Literal behavior also deserves a requirement-level analysis. Table 3 implies about 0.72 explicitly covered requirements per initial instruction, while Literal@1 scores are substantially higher than the approximately 14% suggested by counting only these requirements. This is not necessarily inconsistent: initial functionality, shared implementation effects or partially stated rules may satisfy additional conditions. Literal@4 also receives new explicit feedback, so its 55% to 60% SWE Success for some models does not itself show prompt noncompliance. Finally, Appendix E.4 explicitly accepts early Partial success based on legitimate evidence-driven inference. My concern is therefore not that every such task was discarded, but that retention, repair and exclusion statistics and pre-calibration results are needed to assess the effects of construction. A human Partial baseline would provide further context.

  2. The broad conclusions need clearer qualification by metric, domain and evidence already provided. For three high-performing models on SWE, Partial@4 requirement Score reaches 95.5 to 96.0 versus Full@1's 97.0 to 99.5 (Table 8), but whole-task Success still differs by 7.5 to 12.5 points. Other models retain larger SWE gaps, while research tasks are harder even under Full, whose mean Success is about 49%. These distinctions should inform claims about what feedback can and cannot compensate for. The benchmark makes hidden requirements inferable and mandatory, without a separate measure of harmful over-inference, although incorrect changes can still fail existing checks. The simulated user answers from their knowledge and says they do not know only when the question exceeds it; this limits confirmation of user-unknown rules rather than preventing all clarification. Section 5.6 provides useful reliability evidence: three-run results for two models and an audit of simulator consistency. I would still welcome task-level confidence intervals, broader model replication and additional plan-mode designs before generalizing the conclusions to real deployment.

Questions for authors
  1. How does the Full–Partial gap change when Partial is limited to one submission as well?

  2. Under Literal@1, what are the pass rates for explicitly stated, partially stated and unstated requirements, including those already satisfied initially? How much of Literal@4's improvement comes from newly explicit feedback or changes that satisfy several requirements together?

  3. Is a human success rate available under Partial, even for a small task sample? Given Appendix E.4's acceptance of legitimate early Partial success, please report retention, repair and exclusion counts and the Full–Partial gap before calibration.

  4. Building on the three-run results in Section 5.6, could you report task-bootstrap confidence intervals for the main gaps and broader replication of the plan-mode comparison? How are requests to confirm an inferred rule handled when the owner does versus does not know the answer?

5 comments

Thank you for the detailed review. Weakness 1 asks whether the Full-Partial gap reflects the intended ability or differences in strategy and construction; Weakness 2 asks that the conclusions be qualified by metric, domain and evidence. We ran each control you proposed:

  • W1, Q1 (submission strategy): with one submission under Partial, the gap stays at 54.7-60.9 points, and first-submission Score falls rather than rises.
  • W1, Q2 (Literal): requirement-level pass rates explain why Literal@1 exceeds instruction coverage; most of Literal@4's gains follow feedback that targets the requirement.
  • W1, Q3 (construction and human baseline): we report retention, review and repair statistics, and observations that bear on construction effects.
  • W2, Q4: task-bootstrap CIs, a second harness design, a harmful over-inference screen, and revised claims.

W1, Q1. Does the gap come from early submission under Partial?

No. We reran Partial on all 64 tasks with a single submission for three models. The agent is told that only one submission is allowed, and the limit is enforced; everything else is unchanged.

ModelFull@1 Score / SuccessPartial@1, 4 allowedPartial@1, 1 allowedFull − Partial (1 allowed), Success [95% CI]
GPT-5.6-Luna84.1 / 60.941.6 / 3.140.3 / 0.060.9 [50.0, 71.9]
Kimi-K392.2 / 75.068.8 / 21.962.5 / 18.856.3 [43.8, 68.8]
DeepSeek-V4.1-Flash94.4 / 81.366.3 / 25.065.6 / 26.654.7 [42.2, 67.2]

If agents under Partial submitted early to obtain feedback, removing the later submissions should raise first-submission performance. It does the opposite: Score@1 decreases for all three models, and Success@1 changes by at most 3.1 points. The submission budget does not explain the gap. We will add this control to Section 5.2.

W1, Q2. Literal at the requirement level

Coverage. Of the 320 requirements, 46 (14.4%) are stated or entailed by the initial instruction, matching your estimate of 0.72 per task. We split the remaining 274 into partially stated (117: the instruction specifies a concrete aspect but not the complete criterion) and unstated (157). Re-evaluating all 64 initial workspaces confirms that none of the 320 requirements holds before the agent acts.

Literal@1 pass rate (%)Stated / entailed (46)Partially stated (117)Unstated (157)
DeepSeek-V4.1-Flash56.544.452.9
MiMo-v2.6-Pro67.437.649.0
GLM-5.3-Flash56.531.644.6
Kimi-K347.824.838.2
GPT-5.6-Sol47.826.531.2
MiMo-v2.6-Flash43.524.831.2
GPT-5.6-Terra41.325.626.1
GPT-5.6-Luna41.319.726.1

Why Literal@1 exceeds 14%. Two of the sources you anticipated. (i) Partially stated requirements: the instruction already points at part of the rule, and Literal passes 19.7-44.4% of them. (ii) The Literal prompt does not fully suppress inference: reading project files to locate the requested change exposes evidence of other requirements, and some models act on it. In SWE-partial-structuring, six models added an unrequested subsequent-correction feature at Literal@1; disabling only that feature reduces each first submission from two satisfied requirements to one, with every other requirement unchanged.

Literal is therefore a prompt-level approximation of instruction-only execution that overestimates it, so the Partial-Literal margins we report (15.7-34.1 Score points) are a lower bound on what inference contributes. Our central claim does not rest on Literal: it compares Full and Partial, which share the same prompt. We will state this in Section 5.1.

Where Literal@4's gains come from. Among requirements that fail at Literal@1 and pass at Literal@4, 65.4-77.1% (range over the eight models) first pass in the submission immediately after an observation that targets them, i.e. once feedback has made them explicit. Coordinated revisions are rare: of the 1,301 consecutive-submission transitions in the 512 Literal trajectories, 77 (5.9%) newly satisfy two to five requirements at once. We agree that Literal@4 success does not indicate prompt non-compliance.

W1, Q3. Construction effects and a human baseline

Retention and exclusion. TiGanTrans produced 109 candidate tasks; 64 were retained. The 45 exclusions were made during review and calibration, mainly for environment defects that could not be repaired and for departures from realistic practice that repair could not fix.

Review and repair. Each candidate was assigned to two trained researchers, who reviewed it independently against the four criteria of Section 4.2 and repaired it; a task was accepted only when both agreed that every criterion held. Over the review iterations of the 64 retained tasks, reviewers raised 64 flags on consistency among requirements, evaluator and observations, 46 on the plausibility of the initial instruction, 55 on whether workspace evidence suffices to infer each requirement and determine the required behavior, and 26 on leakage. Every retained task needed human repair, which also shows where automated synthesis currently falls short. We will add these statistics to Appendix E.

Observations that bear on construction effects.

  • The gap is consistent across eight models from five families (53.1-67.2 Success@1 points), and tasks are not unsolvable under Partial: Kimi-K3, MiMo-v2.6-Pro and DeepSeek-V4.1-Flash succeed at the first submission on 21.9-28.1% of them.
  • Every task is solvable from the complete specification: Full@1 Success is 60.9-82.8% for the eight models on the same tasks, workspace and evaluator.
  • The feedback-phase results do not depend on the first-submission gap: the in-round and ahead resolution rates of Table 5 and the failure modes of Section 5.5 describe what agents do after the first submission.
  • The absolute size of the gap is a property of TiGanEval's tasks, not an estimate of naturally occurring gaps; we will state this in Sections 4.3 and 5.2.

Human Partial baseline (pilot). Within the rebuttal period we could recruit two experienced programmers who were not involved in this work. Each completed SWE-config-drift-baselines and the battery-lifetime task Research-pilot-cycle-life-drift under Partial in the same project workspace, scored at the first submission. A person who has solved a task remembers it, so each participant could take each task under only one condition. Mean Score@1 was 60% on the SWE task (0.4 and 0.8) and 80% on the research task (0.6 and 1.0), with one full success in four attempts. On the same two tasks, the eight frontier models averaged 40% and 50% in their first runs, with no full success in 16 attempts.

In follow-up interviews, both participants described first working out what the task was for. On the battery-lifetime task, the participant who satisfied all five requirements recognized that the owner had to decide whether to change the charging protocol and how to schedule further experiments, and then checked the owner's explanation against the measurement records before building on it. This is the behavior missing from the failures in Section 5.5 where the instruction outranked the project. Two participants on two tasks support no quantitative comparison, so we do not add this pilot to the paper. We report it as qualitative evidence that the information needed is present in the workspace, and that the difference lies in whether the instruction is treated as evidence about the owner's goal.

W2. Qualifying the conclusions by metric, domain and evidence

We agree, and will revise the Abstract, Section 5.2 and the Conclusion. The specific distinctions:

  • SWE, three strongest models (Kimi-K3, MiMo-v2.6-Pro, DeepSeek-V4.1-Flash): feedback nearly closes the Score gap (Partial@4 95.5-96.0 vs. Full@1 97.0-99.5), but Success stays 7.5-12.5 points lower, and the recovery costs extra submissions, each a failed delivery the user experienced.
  • SWE, the other five models: the residual Score gap at Partial@4 is 15.5-25.5 points.
  • Research: the residual Score gap at Partial@4 is 17.5-45.9 points.

"Feedback does not close the gap" will be stated for Success; for Score we will state that feedback closes most of it for three models on SWE.

Research tasks are harder under Full. Pass rates of the research checks, pooled over the eight models, show where the difficulty lies:

Research check categoryFull@1 pass (%)Partial@4 pass (%)
Scientific estimation87.543.5
Uncertainty intervals72.431.8
Experimental design, allocation, stopping67.722.4

Full failures concentrate on uncertainty and design. For example, under Full, GPT-5.6-Sol clipped an uncertainty interval to the requested precision and then stopped collecting data because the clipped interval met the target; removing only the clipping turns that check from fail to pass. Partial lowers every category further. Feedback is also limited in research for a structural reason: in the battery-lifetime task under Partial, all eight models kept collecting measurements after the evidence already supported a decision, and later revisions can change the analysis but cannot recover the measurement budget already spent.

Harmful over-inference. We screened 64 Partial trajectories (four SWE and four research tasks, fixed at random, × eight models) for unsupported added rules that cause a verified adverse outcome. Two were confirmed by counterfactual re-evaluation, both from GPT-5.6-Terra: an exact-match restriction that rejected valid snapshot restores, and a reporting restriction that omitted paid calibration measurements. Removing only the added rule raises each from three to four satisfied requirements. Harmful over-inference is present but rare in this sample; we will report it and list a dedicated over-inference task set as future work.

W2, Q4. Uncertainty, harness replication, and clarification

Task-bootstrap CIs. We resample tasks (50,000 paired resamples, stratified by domain), keeping each task's outcomes across conditions together. For all eight frontier models, both the initial gap (Full@1 − Partial@1) and the residual gap (Full@1 − Partial@4) exclude zero for Score and Success. Residual Success gaps range from 15.6 [3.1, 28.1] (Kimi-K3) to 46.9 [34.4, 59.4] (GLM-5.3-Flash); the full table is in our response to Reviewer JSEE and will be added to the appendix. For the two models with three runs, averaging runs within each task before bootstrapping gives residual Success gaps of 36.5 [27.1, 45.8] (GPT-5.6-Luna) and 27.1 [18.8, 35.4] (DeepSeek-V4.1-Flash).

A second harness design. We evaluated an evidence-tracking harness that records requirement hypotheses, links each to project evidence, runs checks that discriminate between competing interpretations, and revises the plan when new evidence contradicts it. Against Partial with the same model, tasks and one-submission budget:

ModelScore@1Δ [95% CI]Success@1Δ [95% CI]
GPT-5.6-Luna40.3 → 42.5+2.2 [−3.4, 7.5]0.0 → 3.1+3.1 [0.0, 7.8]
Kimi-K362.5 → 62.50.0 [−5.6, 5.6]18.8 → 17.2−1.6 [−10.9, 7.8]
DeepSeek-V4.1-Flash65.6 → 70.9+5.3 [0.9, 9.7]26.6 → 26.60.0 [−9.4, 9.4]

DeepSeek-V4.1-Flash gains 13.3 [4.2, 23.3] Score points on research tasks, so the benchmark registers harness improvements when they occur. No model shows a Success@1 change whose interval excludes zero. We will state that the two harness designs we tested do not close the gap, rather than that harness engineering cannot.

Confirming an inferred rule. The simulated owner answers from its supplied knowledge and its earlier messages (Appendix F.4). If the rule is within its knowledge, it confirms or corrects it; otherwise it says it does not know and suggests checking the project. Clarification is therefore always available, but it returns only what the owner knows, which is the situation the benchmark targets.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We thank the reviewers for their reviews and the AC for their efforts. We summarize our rebuttal and revisions below.

We appreciate that the reviewers acknowledged our core conceptual shift — treating user instructions as fallible evidence rather than ground-truth objectives — alongside our paired design and diagnostic depth:

  • Reviewers JSEE and pdmA highlighted the importance and clean formalization of outcome alignment — separating stated, inferred, and true requirements, motivated by SWE-chat where 56% of initial instructions omit needed requirements and 52% of useful follow-ups are project-inferable. Unlike clarification settings with a right user goal, here users lack system knowledge and may propose mechanisms contradicted by project records.
  • All three reviewers valued our paired Full, Partial, and Literal protocol over a shared workspace and frozen evaluator; Reviewer qv3Z noted separating pre-feedback resolution from post-feedback repair (Table 5) had not appeared in earlier interactive benchmarks, and Reviewer pdmA commended this careful measurement as rare.
  • All three reviewers appreciated our trajectory audit showing 72% of failures stem from mishandling workspace evidence — e.g., patching symptoms without recovering the underlying rule or ignoring violations printed by replays.

Summary of Key Concerns and Our Responses

  1. Novelty over implicit-requirement benchmarks and validity of "true requirements" (raised by JSEE)

    Our Response: Implicit Intelligence and SWE-RPG assume a correct user goal and score single-shot runs without consequence feedback or a full-specification control (Table 1). In TiGanEval, missing requirements lie outside user knowledge and user proposals can be wrong (10% of failures obey mistaken user beliefs over project records), so neither asking nor faithful completion suffices. Requirements come from independent tasks, pass dual-researcher verification so workspace evidence uniquely fixes behavior, and use implementation-agnostic checks on held-out workloads.

  2. Controls on the Full-Partial gap: submission budget, prompt wording, and Literal behavior (raised by qv3Z, pdmA)

    Our Response: Full and Partial share the identical system prompt. Across all 64 tasks (Appendix L): (1) limiting Partial to one submission keeps the Success@1 gap at 54.7-60.9 points while Score@1 slightly drops, ruling out strategic early submission; (2) a Neutral prompt leaves Success@1 at 3.1-18.8% (vs. Full 60.9-81.3%), attributing the deficit to the models; and (3) 0/320 requirements hold initially, explaining Literal@1 via partially stated rules (117/320) and incidental inference upon reading files, with 65.4-77.1% of Literal@4 gains following targeted feedback.

  3. Pipeline review statistics, gap severity, same-family bias, and statistical reliability (raised by qv3Z, JSEE, pdmA)

    Our Response: From 109 candidates, 64 were retained after dual-researcher review and repair across four criteria (Appendix E.5). The gap exceeds 56 Success points even with 2/5 requirements initially stated, and a pilot human check confirms workspace discoverability. Claude Sonnet 5 shows no same-family advantage from the Claude Opus 5 pipeline (lowest Partial Score@1 among nine models; 63.6-point Success@1 gap), and 50,000-resample paired bootstrap 95% CIs exclude zero for all eight frontier models at @1 and @4 (Appendix I.1).

  4. Stronger harnesses, research difficulty vs. noise, and claim qualifications (raised by qv3Z, JSEE, pdmA)

    Our Response: An evidence-tracking harness (hypothesis tracking, evidence linking, discriminative checks, plan revision) lifts DeepSeek-V4.1-Flash's research Score@1 by +13.3 points, yet Success@1 stays at 3.1-26.6% with all Success-change CIs spanning zero (Appendix L.3). On research tasks, 91/98 Full failures reflect genuine uncertainty/design errors rather than noise, paired CIs exclude zero, and where Full succeeds 3/3 times, Partial@1 succeeds in 1/24 runs (Appendix I.2). We also screened 64 runs for over-inference (two cases) and qualified claims across metrics, domains, and harnesses.

Conclusion

Our work challenges the assumption that instructions are complete objectives — showing that agents should treat instructions as fallible evidence weighed against the environment — and contributes TiGanEval and TiGanTrans, with all new experiments integrated into the revised paper. Reviewer JSEE (Confidence 4) found "the revised evidence convincing" and raised their rating from 2 to 3, and Reviewer pdmA confirmed "all three concerns are well addressed" and maintained 3. While we regret not having exchanged with Reviewer qv3Z during the discussion period, qv3Z noted only minor, fixable soundness concerns, and we believe our responses address all of them — every requested control, breakdown, bootstrap interval, harness, and qualification has been completed and added. We thank the reviewers and AC again for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.