- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a benchmark for evaluating and understanding how models handle ambiguous user intent when given prior coding history.
Critical strengths
- The problem tackled is important as coding tasks become more and more ambiguous.
- The overall idea of having prior sessions and retrieving that matches realistic workflows.
- The experimental results and findings provide useful insights.
Critical weaknesses
- This benchmark, unfortunately, already seems highly saturated. Even the hardest problems for ES are already at 75%+.
- Using humaneval problems as the source seems incredibly outdated; there are a plethora of newer coding benchmarks that feature more realistic and more complex tasks, and I do not understand why humaneval was chosen.
- The entire section on interactive dialogue generation is very difficult to read. There are a significant number of different agents at play here and it's very unclear how they all interact.
- I'm not sure that cross session history improving task completion and reducing clarification is a novel claim? There have been several works related to this and it's unclear to me what makes this work differentiated.
- Table 1: ES,FT-ES,TTC are not defined; the table is not readable in isolation. Also, a nitpick but the model ordering seems random; it is neither sorted by best score nor by date of release.
Also, I believe that the following two citations are relevant and missing.
ToM-SWE ( https://icml.cc/virtual/2026/poster/65025 ) covers a similar premise on ambiguous user intent with a user simulator. In fact, I would go so far as to say that there are questions of novelty associated here. EDIT-Bench (https://iclr.cc/virtual/2026/oral/10010553) is a coding benchmark that explicitly includes ambiguous user instructions.
Questions for authors
- Why choose humaneval over other coding problem sources?
- What is the differentiation of this paper?
7 comments
W2 & Q1: HumanEval is chosen by design to reduce coding-difficulty confounds
Task completion depends on both implementing and debugging code, and understanding the intended requirements. Harder coding tasks amplify the former, making ambiguity resolution harder to assess separately. We therefore chose relatively simple HumanEval tasks to reduce this confound. HumanEval also provides self-contained specifications and executable tests, allowing us to control which requirements are omitted while keeping the underlying task fixed.
HumanEval supplies the base tasks; the realism of CAPA lies in how requests are made ambiguous. The six ambiguity mechanisms are grounded in real coding conversations from WildChat (Sec. 1), and annotators recognize them in third-party-written requests (Fleiss κ = 0.66 on 200 HumanEvalComm requests, Sec. 4.1). In an additional human evaluation of 48 generated dialogues (see our response to R2), 62.5% of ratings were at least 4/5 for naturalness and ambiguity realism (mean 3.60), and 88.5% were at least 4/5 for intent recoverability (mean 4.39).
On HumanEval, we compare complete specifications (Original) with ambiguous requests without history (No). ES and FT-ES are percentages; TTC is mean completion turns.
| Model | Condition | ES ↑ | FT-ES ↑ | TTC ↓ |
|---|---|---|---|---|
| GPT-5.5 | Original | 100.0 | 89.0 | 1.12 |
| No | 74.3 | 2.3 | 4.42 | |
| No−Original | −25.7 | −86.7 | +3.30 | |
| DeepSeek V4 Pro | Original | 100.0 | 91.5 | 1.10 |
| No | 76.3 | 15.3 | 4.09 | |
| No−Original | −23.7 | −76.2 | +2.99 |
With complete specifications, both models reach 100% ES and finish in about 1.1 turns, indicating little difficulty implementing these tasks. Introducing ambiguity lowers FT-ES by 86.7 and 76.2 pp and adds about three completion turns. Because the underlying coding tasks are unchanged, this contrast makes the difficulty introduced by missing requirements more visible.
We extended the same pipeline to BigCodeBench-Hard v0.1.4. We constructed 30 user groups with five histories and five held-out tasks each, keeping history and test source tasks separate. We compare complete specifications (Original), ambiguous requests without history (No), and ambiguous requests with Same-user history (Same). All models use the same GPT-5.5-generated histories; each tested model also generates the live user replies. We evaluate up to twelve turns and report the eight-turn cutoff of the same trajectories. FT-ES is reported once because the first response is unchanged. ES and FT-ES are percentages. TTC assigns the turn limit to unfinished sessions. DS and GLM denote DeepSeek V4 Pro and GLM-5.2.
| Model | Condition | FT-ES ↑ | ES@8 ↑ | TTC@8 ↓ | ES@12 ↑ | TTC@12 ↓ |
|---|---|---|---|---|---|---|
| GPT-5.5 | Original | 29.3 | 95.3 | 2.41 | 96.7 | 2.57 |
| No | 0.7 | 42.0 | 7.23 | 73.3 | 8.97 | |
| Same | 2.0 | 53.3 | 6.67 | 72.0 | 8.19 | |
| DS | Original | 40.0 | 82.7 | 3.19 | 84.0 | 3.88 |
| No | 1.3 | 30.7 | 7.05 | 42.7 | 9.62 | |
| Same | 2.7 | 29.3 | 6.95 | 36.7 | 9.67 | |
| GLM | Original | 32.0 | 80.0 | 3.24 | 81.3 | 4.01 |
| No | 2.0 | 22.0 | 7.35 | 32.0 | 10.30 | |
| Same | 2.7 | 33.3 | 7.12 | 42.0 | 9.63 |
On this harder base, history effects are small and inconsistent across models, because coding difficulty is confounded with ambiguity resolution. Inspecting the interaction logs, we found that models spent substantial effort debugging, and in many sessions the main task was not completed within the turn budget. Consistent with this, even with complete specifications, first-turn success is only 29.3–40.0% (versus 89.0–91.5% on HumanEval) and sessions need 2.57–4.01 turns. This repair work is separate from ambiguity resolution: in GPT-5.5's twelve-turn breakdown below, execution-feedback repair takes 2.49 turns without history and 1.97 turns with Same-user history. The effect of ambiguity resolution therefore cannot be isolated on this base, which is why we chose HumanEval, where the underlying tasks are reliably solved once requirements are clear.
GPT-5.5's twelve-turn breakdown makes this mixture explicit:
| Turns | No | Same | Reduction |
|---|---|---|---|
| Requirement clarification | 5.48 | 5.22 | 0.26 |
| Execution-feedback repair | 2.49 | 1.97 | 0.51 |
| TTC@12 | 8.97 | 8.19 | 0.77 |
Of the 0.77-turn reduction, 0.51 comes from execution-feedback repair and 0.26 from requirement clarification. A lower TTC on harder tasks therefore cannot be attributed entirely to better ambiguity resolution. This supports our choice of HumanEval to reduce coding-difficulty confounds. The extension uses different API snapshots, so cross-source differences are not an isolated estimate of task difficulty.
We will clarify this design choice and add the harder-task results and turn breakdown to the paper.
W4 & Q2: CAPA evaluates recurring user ambiguity across coding tasks
Thank you for recommending ToM-SWE and EDIT-Bench. We will cite and discuss both works in Related Work.
What this paper studies and what is new. Related work falls into two lines. The first studies ambiguity in coding tasks and how models resolve it. Ambig-SWE, HumanEvalComm, and ClarifyCodeBench examine how models identify missing requirements and recover them through clarification. EDIT-Bench evaluates context-dependent code-editing instructions, where the current code context helps determine the intended edits. The second studies user modeling and personalization. ToM-SWE proposes a user-modeling method with persistent user memory. It also introduces the Stateful SWE benchmark, which supports intent inference from user histories and provides a profile-conditioned user simulator. Training Proactive and Personalized LLM Agents (PPP) trains agents to follow users’ interaction preferences. CAPA connects these two lines through a specific question: can models learn a user’s recurring ambiguity and resolution patterns from resolved histories and apply them to new coding tasks with fewer or no clarification questions? Its contribution is a controlled benchmark that fixes these patterns across tasks and varies the historical evidence available to the model. We will revise the contribution statement to reflect this focus.
Comparison with related work. The table organizes the comparison along three dimensions: task setting (columns 1–3), user modeling (columns 4–6), and evaluation design (columns 7–8). ✓ = included in the reported setting; ✗ = not included. CAPA’s history-control entry includes the control added in this rebuttal.
| Work | Ambiguous coding/API requests | Execution-based correctness | User clarification | Cross-session dialogue history | User preferences/profiles | Explicitly fixed cross-task ambiguity mechanisms | Mechanism-aware history controls | Interaction evaluation |
|---|---|---|---|---|---|---|---|---|
| LaMP | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
| LongMemEval | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ |
| PersonaMem | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ |
| PRefine | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ |
| HumanEvalComm | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ |
| ClarifyCodeBench | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ |
| EDIT-Bench | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Ambig-SWE | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ |
| PPP | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✓ |
| Stateful SWE (ToM-SWE) | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |
| CAPA (Ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Task setting. CAPA tests whether models can recover a new task's missing requirements from the same user's resolved history. In existing benchmarks, history is either absent (clarification benchmarks and EDIT-Bench) or conveys the user's preferences (Stateful SWE). In CAPA, history and new tasks are built from the same user-level ambiguity–resolution assignment (Sec. 4.2), so history shows how this user's omitted requirements were resolved before, and models can infer the new task's missing requirements from it.
User modeling. CAPA treats ambiguity as a property of the user. Stateful SWE represents a user by interaction styles and coding preferences, and creates ambiguity separately for each issue. In CAPA, each user additionally has a fixed ambiguity mechanism (how they omit or obscure requirements) and a fixed resolution pattern (how those requirements are resolved), which stay the same across tasks (Sec. 3). The same user therefore omits the same kinds of information, and resolves them in the same way, in every task.
Evaluation design. CAPA asks where the benefit of history comes from: generic dialogue examples, or evidence about this user's ambiguity patterns. The compared works do not report a control that separates the two. Because CAPA fixes each user's patterns, we can build a control history from other users whose ambiguity types do not overlap with the target's, and compare it with Same-user history on the same tasks. Same-user history raises FT-ES over this control by 20.0 pp for GPT-5.5 and 8.3 pp for DeepSeek V4 Pro (see our responses to R1 and R2), an additional gain from relevant evidence beyond generic examples.
W5: We will make Table 1 self-contained
We will make Table 1 self-contained by defining ES, FT-ES, and TTC in its caption, and order the models by release date.
Thank you for recognizing the importance of ambiguous intent, the realism of using prior sessions, and the experimental insights. We will upload a revision with changes highlighted in blue.
W1: First-turn ambiguity resolution remains far from saturated
High ES is expected on our relatively simple coding base. We chose HumanEval to separate ambiguity from coding difficulty: GPT-5.5 and DeepSeek achieve 100% ES with complete specifications (Table 6). ES allows eight turns, so models can also succeed by clarifying missing requirements rather than inferring them from history.
The benchmark tests both eventual correctness and whether history avoids repeated clarification. ES measures the former; FT-ES and TTC measure success before clarification and completion cost. The comparison below shows that first-turn success and completion cost remain far from their ambiguity-free levels, even with Same-user history:
Original = complete HumanEval specification; No/Same = CAPA without/with Same-user history; DS = DeepSeek V4 Pro.
Across all models, the best Same-user FT-ES is only 60.3%. On complex sessions, it falls to 7.0–27.2%, with 3.61–4.89 turns needed for completion (Table 2 in paper). These results show substantial remaining difficulty in resolving ambiguity without additional interaction.
We will clarify this joint evaluation of correctness and interaction cost in the main text.
W3: We will clarify the pipeline
We agree that the components and their interactions need a clearer explanation. The pipeline has two stages: data generation creates the user histories and held-out tasks (Sec. 4); evaluation tests whether a model uses those histories to resolve new requests (Secs. 3 and 5).
Generation uses a Conversation LLM, a user simulator, and an execution environment. The simulator contains a GT-aware planner, an expression agent, and a generation judge. We inject ambiguity by omitting or obscuring requirements according to each user's fixed ambiguity types and resolution patterns, then phrase the request in their communication style. The Conversation LLM asks questions or submits code. The environment tests the code, and the planner guides the expression agent's next user reply. The generation judge,
J_gen, is an LLM that uses task information, submitted code, and execution results to decide whether the generated session is complete. These user patterns remain fixed across sessions; histories retain only user–assistant messages.Evaluation replaces the Conversation LLM with the model under test. The simulator retains its planner and expression agent, but success is determined by the execution-only judge
Jthrough hidden tests in the IDE, not byJ_gen. The tested model receives history, the new request, user replies, and execution feedback. GT and planner outputs remain on the user side; hidden tests are not exposed to the tested model.We will update the paper to make this more clear.