acceptodds
Under review as a conference paper at ICLR 2027

Fewer Clarifications, Better Code: Benchmarking Cross-Session Personalized Ambiguity Adaptation in Coding Assistants

Abstract

AI-assisted coding increasingly translates informal user intent into executable software, yet coding requests often contain ambiguities that recur in user-specific ways across tasks and sessions. Existing disambiguation methods typically address each ambiguous request in isolation within the current coding session, often through eliciting additional clarification. However, whether resolved session history from the same user can serve as memory for resolving recurring personalized ambiguity in a newly opened session remains underexplored. We formulate personalized ambiguity adaptation as a new task: given a user's previously resolved coding sessions and a new ambiguous request, an assistant should identify the recurring ambiguity pattern, produce the intended executable solution, and minimize clarification. To benchmark this task, we introduce CAPA, which characterizes personalized coding ambiguity through six mechanisms and injects these mechanisms into unambiguous executable tasks using a controlled three-stage generation pipeline. CAPA contains 600 coding sessions across 60 balanced user–ambiguity cells, including 300 held-out evaluation sessions. We evaluate 12 recent LLMs under no-history and same-user-history conditions using executable success, first-turn success, and turns-to-completion. Our analyses examine task difficulty, user identity, and memory-based history use, and we further propose same-user history gating as a lightweight inference-time method. CAPA provides a foundation for developing long-term coding assistants that better align generated code with user intent while reducing repeated clarification.

Primary area:
datasets and benchmarks
Keywords:
Coding Assistants, Personalized Ambiguity Resolution and Clarification, Cross-Session Adaptation, Long-Term Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ho3uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a benchmark for evaluating and understanding how models handle ambiguous user intent when given prior coding history.

Critical strengths
  1. The problem tackled is important as coding tasks become more and more ambiguous.
  2. The overall idea of having prior sessions and retrieving that matches realistic workflows.
  3. The experimental results and findings provide useful insights.
Critical weaknesses
  1. This benchmark, unfortunately, already seems highly saturated. Even the hardest problems for ES are already at 75%+.
  2. Using humaneval problems as the source seems incredibly outdated; there are a plethora of newer coding benchmarks that feature more realistic and more complex tasks, and I do not understand why humaneval was chosen.
  3. The entire section on interactive dialogue generation is very difficult to read. There are a significant number of different agents at play here and it's very unclear how they all interact.
  4. I'm not sure that cross session history improving task completion and reducing clarification is a novel claim? There have been several works related to this and it's unclear to me what makes this work differentiated.
  5. Table 1: ES,FT-ES,TTC are not defined; the table is not readable in isolation. Also, a nitpick but the model ordering seems random; it is neither sorted by best score nor by date of release.

Also, I believe that the following two citations are relevant and missing.

ToM-SWE ( https://icml.cc/virtual/2026/poster/65025 ) covers a similar premise on ambiguous user intent with a user simulator. In fact, I would go so far as to say that there are questions of novelty associated here. EDIT-Bench (https://iclr.cc/virtual/2026/oral/10010553) is a coding benchmark that explicitly includes ambiguous user instructions.

Questions for authors
  1. Why choose humaneval over other coding problem sources?
  2. What is the differentiation of this paper?
7 comments

Thank you for recognizing the importance of ambiguous intent, the realism of using prior sessions, and the experimental insights. We will upload a revision with changes highlighted in blue.

W1: First-turn ambiguity resolution remains far from saturated

High ES is expected on our relatively simple coding base. We chose HumanEval to separate ambiguity from coding difficulty: GPT-5.5 and DeepSeek achieve 100% ES with complete specifications (Table 6). ES allows eight turns, so models can also succeed by clarifying missing requirements rather than inferring them from history.

The benchmark tests both eventual correctness and whether history avoids repeated clarification. ES measures the former; FT-ES and TTC measure success before clarification and completion cost. The comparison below shows that first-turn success and completion cost remain far from their ambiguity-free levels, even with Same-user history:

Original = complete HumanEval specification; No/Same = CAPA without/with Same-user history; DS = DeepSeek V4 Pro.

ModelMetricOriginalNoSame
GPT-5.5FT-ES ↑89.0%2.3%31.0%
TTC ↓1.124.422.97
DSFT-ES ↑91.5%15.3%17.7%
TTC ↓1.104.093.67

Across all models, the best Same-user FT-ES is only 60.3%. On complex sessions, it falls to 7.0–27.2%, with 3.61–4.89 turns needed for completion (Table 2 in paper). These results show substantial remaining difficulty in resolving ambiguity without additional interaction.

ModelFT-ES ↑TTC ↓
GPT-5.521.1%3.97
DS7.0%4.89
GLM27.2%3.61

We will clarify this joint evaluation of correctness and interaction cost in the main text.

W3: We will clarify the pipeline

We agree that the components and their interactions need a clearer explanation. The pipeline has two stages: data generation creates the user histories and held-out tasks (Sec. 4); evaluation tests whether a model uses those histories to resolve new requests (Secs. 3 and 5).

Generation uses a Conversation LLM, a user simulator, and an execution environment. The simulator contains a GT-aware planner, an expression agent, and a generation judge. We inject ambiguity by omitting or obscuring requirements according to each user's fixed ambiguity types and resolution patterns, then phrase the request in their communication style. The Conversation LLM asks questions or submits code. The environment tests the code, and the planner guides the expression agent's next user reply. The generation judge, J_gen, is an LLM that uses task information, submitted code, and execution results to decide whether the generated session is complete. These user patterns remain fixed across sessions; histories retain only user–assistant messages.

Evaluation replaces the Conversation LLM with the model under test. The simulator retains its planner and expression agent, but success is determined by the execution-only judge J through hidden tests in the IDE, not by J_gen. The tested model receives history, the new request, user replies, and execution feedback. GT and planner outputs remain on the user side; hidden tests are not exposed to the tested model.

We will update the paper to make this more clear.

W2 & Q1: HumanEval is chosen by design to reduce coding-difficulty confounds

Task completion depends on both implementing and debugging code, and understanding the intended requirements. Harder coding tasks amplify the former, making ambiguity resolution harder to assess separately. We therefore chose relatively simple HumanEval tasks to reduce this confound. HumanEval also provides self-contained specifications and executable tests, allowing us to control which requirements are omitted while keeping the underlying task fixed.

HumanEval supplies the base tasks; the realism of CAPA lies in how requests are made ambiguous. The six ambiguity mechanisms are grounded in real coding conversations from WildChat (Sec. 1), and annotators recognize them in third-party-written requests (Fleiss κ = 0.66 on 200 HumanEvalComm requests, Sec. 4.1). In an additional human evaluation of 48 generated dialogues (see our response to R2), 62.5% of ratings were at least 4/5 for naturalness and ambiguity realism (mean 3.60), and 88.5% were at least 4/5 for intent recoverability (mean 4.39).

On HumanEval, we compare complete specifications (Original) with ambiguous requests without history (No). ES and FT-ES are percentages; TTC is mean completion turns.

ModelConditionES ↑FT-ES ↑TTC ↓
GPT-5.5Original100.089.01.12
No74.32.34.42
No−Original−25.7−86.7+3.30
DeepSeek V4 ProOriginal100.091.51.10
No76.315.34.09
No−Original−23.7−76.2+2.99

With complete specifications, both models reach 100% ES and finish in about 1.1 turns, indicating little difficulty implementing these tasks. Introducing ambiguity lowers FT-ES by 86.7 and 76.2 pp and adds about three completion turns. Because the underlying coding tasks are unchanged, this contrast makes the difficulty introduced by missing requirements more visible.

We extended the same pipeline to BigCodeBench-Hard v0.1.4. We constructed 30 user groups with five histories and five held-out tasks each, keeping history and test source tasks separate. We compare complete specifications (Original), ambiguous requests without history (No), and ambiguous requests with Same-user history (Same). All models use the same GPT-5.5-generated histories; each tested model also generates the live user replies. We evaluate up to twelve turns and report the eight-turn cutoff of the same trajectories. FT-ES is reported once because the first response is unchanged. ES and FT-ES are percentages. TTC assigns the turn limit to unfinished sessions. DS and GLM denote DeepSeek V4 Pro and GLM-5.2.

ModelConditionFT-ES ↑ES@8 ↑TTC@8 ↓ES@12 ↑TTC@12 ↓
GPT-5.5Original29.395.32.4196.72.57
No0.742.07.2373.38.97
Same2.053.36.6772.08.19
DSOriginal40.082.73.1984.03.88
No1.330.77.0542.79.62
Same2.729.36.9536.79.67
GLMOriginal32.080.03.2481.34.01
No2.022.07.3532.010.30
Same2.733.37.1242.09.63

On this harder base, history effects are small and inconsistent across models, because coding difficulty is confounded with ambiguity resolution. Inspecting the interaction logs, we found that models spent substantial effort debugging, and in many sessions the main task was not completed within the turn budget. Consistent with this, even with complete specifications, first-turn success is only 29.3–40.0% (versus 89.0–91.5% on HumanEval) and sessions need 2.57–4.01 turns. This repair work is separate from ambiguity resolution: in GPT-5.5's twelve-turn breakdown below, execution-feedback repair takes 2.49 turns without history and 1.97 turns with Same-user history. The effect of ambiguity resolution therefore cannot be isolated on this base, which is why we chose HumanEval, where the underlying tasks are reliably solved once requirements are clear.

GPT-5.5's twelve-turn breakdown makes this mixture explicit:

TurnsNoSameReduction
Requirement clarification5.485.220.26
Execution-feedback repair2.491.970.51
TTC@128.978.190.77

Of the 0.77-turn reduction, 0.51 comes from execution-feedback repair and 0.26 from requirement clarification. A lower TTC on harder tasks therefore cannot be attributed entirely to better ambiguity resolution. This supports our choice of HumanEval to reduce coding-difficulty confounds. The extension uses different API snapshots, so cross-source differences are not an isolated estimate of task difficulty.

We will clarify this design choice and add the harder-task results and turn breakdown to the paper.

W4 & Q2: CAPA evaluates recurring user ambiguity across coding tasks

Thank you for recommending ToM-SWE and EDIT-Bench. We will cite and discuss both works in Related Work.

What this paper studies and what is new. Related work falls into two lines. The first studies ambiguity in coding tasks and how models resolve it. Ambig-SWE, HumanEvalComm, and ClarifyCodeBench examine how models identify missing requirements and recover them through clarification. EDIT-Bench evaluates context-dependent code-editing instructions, where the current code context helps determine the intended edits. The second studies user modeling and personalization. ToM-SWE proposes a user-modeling method with persistent user memory. It also introduces the Stateful SWE benchmark, which supports intent inference from user histories and provides a profile-conditioned user simulator. Training Proactive and Personalized LLM Agents (PPP) trains agents to follow users’ interaction preferences. CAPA connects these two lines through a specific question: can models learn a user’s recurring ambiguity and resolution patterns from resolved histories and apply them to new coding tasks with fewer or no clarification questions? Its contribution is a controlled benchmark that fixes these patterns across tasks and varies the historical evidence available to the model. We will revise the contribution statement to reflect this focus.

Comparison with related work. The table organizes the comparison along three dimensions: task setting (columns 1–3), user modeling (columns 4–6), and evaluation design (columns 7–8). ✓ = included in the reported setting; ✗ = not included. CAPA’s history-control entry includes the control added in this rebuttal.

WorkAmbiguous coding/API requestsExecution-based correctnessUser clarificationCross-session dialogue historyUser preferences/profilesExplicitly fixed cross-task ambiguity mechanismsMechanism-aware history controlsInteraction evaluation
LaMP✗✗✗✗✓✗✗✗
LongMemEval✗✗✗✓✓✗✗✗
PersonaMem✗✗✗✓✓✗✗✗
PRefine✓✗✗✓✓✗✗✗
HumanEvalComm✓✓✓✗✗✗✗✓
ClarifyCodeBench✓✓✓✗✗✗✗✓
EDIT-Bench✓✓✗✗✗✗✗✗
Ambig-SWE✓✓✓✗✗✗✗✓
PPP✓✓✓✗✓✗✗✓
Stateful SWE (ToM-SWE)✓✓✓✓✓✗✗✓
CAPA (Ours)✓✓✓✓✓✓✓✓

Task setting. CAPA tests whether models can recover a new task's missing requirements from the same user's resolved history. In existing benchmarks, history is either absent (clarification benchmarks and EDIT-Bench) or conveys the user's preferences (Stateful SWE). In CAPA, history and new tasks are built from the same user-level ambiguity–resolution assignment (Sec. 4.2), so history shows how this user's omitted requirements were resolved before, and models can infer the new task's missing requirements from it.

User modeling. CAPA treats ambiguity as a property of the user. Stateful SWE represents a user by interaction styles and coding preferences, and creates ambiguity separately for each issue. In CAPA, each user additionally has a fixed ambiguity mechanism aia_i (how they omit or obscure requirements) and a fixed resolution pattern rir_i (how those requirements are resolved), which stay the same across tasks (Sec. 3). The same user therefore omits the same kinds of information, and resolves them in the same way, in every task.

Evaluation design. CAPA asks where the benefit of history comes from: generic dialogue examples, or evidence about this user's ambiguity patterns. The compared works do not report a control that separates the two. Because CAPA fixes each user's patterns, we can build a control history from other users whose ambiguity types do not overlap with the target's, and compare it with Same-user history on the same tasks. Same-user history raises FT-ES over this control by 20.0 pp for GPT-5.5 and 8.3 pp for DeepSeek V4 Pro (see our responses to R1 and R2), an additional gain from relevant evidence beyond generic examples.

W5: We will make Table 1 self-contained

We will make Table 1 self-contained by defining ES, FT-ES, and TTC in its caption, and order the models by release date.

Reviewer_7qZMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CAPA, a benchmark for studying whether coding assistants can use previous interactions with the same user to resolve recurring ambiguities in new requests. It contains 600 simulated coding sessions, with resolved histories and held-out tasks organized around six ambiguity mechanisms. The evaluation covers 12 models and measures executable correctness, first-turn success, and turns to completion.

Critical strengths

The benchmark addresses a useful problem that ordinary coding evaluations largely miss. An assistant may eventually produce correct code while repeatedly asking a returning user to explain the same assumptions. CAPA makes this measurable along with correctness. There is definitely value in establishing this type of evaluation, even without a new memory architecture etc.

I also appreciate the selection of a proper baseline which is shuffled histories. Shuffled histories test whether improvements actually depend on user identity, while the memory-system comparisons show that organizing history alone does not automatically make it more useful. These analyses make the benchmark more informative than a simple comparison with and without history.

Critical weaknesses

My main concern is whether the improvements come from learning about a specific user or from seeing examples of similar conversations. Histories from other users already provide much of the benefit. For GPT-5.5, these shuffled histories even lead to slightly higher final success than the correct user’s history, although the correct history helps it succeed on the first attempt and finish faster. Using the correct user’s history adds more value for GLM-5.2, so the benefit depends on the model. The authors acknowledge that different histories may contain similar types of ambiguity. I would like a comparison that accounts for those similarities more carefully. This would help show whether the model learns what a particular user means or simply picks up common patterns in the generated conversations. That said, the value of a "query log" is still established without a doubt.

Questions for authors

Was a subset of generated histories independently checked by humans for realistic ambiguity and recoverability of the intended interpretation?

Can you provide paired uncertainty estimates or repeated runs for the main comparisons, particularly the small GPT-5.6-Sol regression?

3 comments

Thank you for recognizing the value of measuring repeated clarification alongside correctness, and the history and memory-system comparisons. We will upload a revision with changes highlighted in blue.

W1: Same-user history adds gains beyond generic examples

We agree that similar dialogue examples may account for some of the gains. An additional control is needed to separate this benefit from adaptation to the target user. The original Table 3 compares No history (C0), shuffled history (C1), and Same-user history (C3). C1 is not purely generic: other users can share the target's recurring ambiguity patterns. In our task, personalization means adapting to the target user's own recurring ambiguity and resolution patterns, which persist across that user's sessions (Sec. 3). Other users may share some of these ambiguity types, so shuffled history can contain evidence relevant to the target user. We therefore added C2: five other-user sessions with no overlap in the target's ambiguity types. This retains the dialogue context but removes target-type overlap. C2−C0 measures the generic context gain, C1−C2 estimates the contribution of target-relevant ambiguity types within shuffled history, and C3−C2 measures the Same-user gain beyond generic examples.

GPT-5.5 and DeepSeek API snapshots changed after submission, so we reran all Table 3 conditions and C2 for GPT-5.5 and DeepSeek V4 Pro. All conditions use the same 12 users with five held-out sessions each (60 sessions per model–condition pair). Prompts, history format, model parameters, and the eight-turn protocol remain unchanged. All entries below come from this rerun, not the original Table 3.

ES is success within eight turns and FT-ES is first-turn success before clarification, both in %. TTC is mean completion turns, assigning eight to unfinished sessions. DS denotes DeepSeek V4 Pro, and pp denotes percentage points.

ConditionMetricGPT-5.5DS
C0: No historyES ↑95.075.0
FT-ES ↑18.313.3
TTC ↓3.053.75
C2: No type overlapES ↑90.080.0
FT-ES ↑33.315.0
TTC ↓2.653.63
C1: Mixed-typeES ↑93.381.7
FT-ES ↑43.321.7
TTC ↓2.273.37
C3: Same-userES ↑95.086.7
FT-ES ↑53.323.3
TTC ↓1.953.00
Total history gain: C3−C0ΔFT-ES (pp)+35.0+10.0
Generic gain: C2−C0ΔFT-ES (pp)+15.0+1.7
Relevant-type increment in mixed history: C1−C2ΔFT-ES (pp)+10.0+6.7
Additional Same-user increment: C3−C1ΔFT-ES (pp)+10.0+1.7
Personalized gain over generic examples: C3−C2ΔFT-ES (pp)+20.0+8.3
Personalized share: (C3−C2)/(C3−C0)%57.183.3
  • Context gain. Even without target-type overlap, solved dialogues improve FT-ES (C2−C0) by 15.0 pp for GPT-5.5 and 1.7 pp for DeepSeek.

  • Personalized gain. Same-user history improves FT-ES over the generic control (C3−C2) by 20.0 pp for GPT-5.5 and 8.3 pp for DeepSeek. These gains account for 57.1% and 83.3% of the total history gain, so user-relevant patterns contribute more than generic examples for both models. Shuffled history also exceeds C2 for both models, so its benefit is not purely generic. The smaller absolute gains for DeepSeek reflect its weaker use of history overall: in the main results (Table 1), history raises its FT-ES by only 2.3 pp, versus 28.7 pp for GPT-5.5.

We will rewrite the discussion of Table 3 to report the generic context gain and the personalized gain separately, and update Table 3 with C2.

Q2: Repeated runs and uncertainty estimates

We repeated the main history comparisons to assess their run-to-run variability, with an additional test of the GPT-5.6-Sol regression you highlight. Below, we report DeepSeek's ten-run results, repeated results for GPT-5.5 and DeepSeek V4 Pro, and the targeted GPT-5.6-Sol results.

To measure run-to-run variability with fixed inputs, we ran DeepSeek V4 Pro ten times per condition on ten fixed tasks from users T02/T03, keeping histories and parameters fixed. The table reports mean ± sample standard deviation across the ten runs, followed by the approximate 95% t CI for the mean [df=9]. ES and FT-ES are percentages (SD in percentage points); TTC is in turns.

ConditionES ↑FT-ES ↑TTC ↓
No history69.00 ± 12.87 [59.80, 78.20]9.00 ± 3.16 [6.74, 11.26]4.47 ± 0.70 [3.97, 4.97]
Shuffled80.00 ± 9.43 [73.26, 86.74]10.00 ± 0.00 [10.00, 10.00]3.79 ± 0.48 [3.45, 4.13]
Same-user84.00 ± 8.43 [77.97, 90.03]11.00 ± 3.16 [8.74, 13.26]3.44 ± 0.40 [3.16, 3.72]

Same-user TTC has relatively low run-to-run variation (SD 0.40 turns) and a mean CI of [3.16, 3.72], although ES still fluctuates. The zero-width shuffled FT-ES interval reflects identical values in these ten runs, not certainty on new tasks.

We also repeated No history (C0), shuffled history (C1), and Same-user history (C3) three times for GPT-5.5 and DeepSeek V4 Pro with the current API versions and unchanged evaluation settings. This experiment uses a separate subset of 50 held-out sessions from ten users, different from the 60-session subset in W1, so absolute values differ from the W1 table. Shuffled histories are resampled with seeds 0, 1, and 2; the other conditions keep identical inputs across runs. The table reports mean ± sample standard deviation across three runs.

ModelConditionES ↑FT-ES ↑TTC ↓
GPT-5.5C0: No history89.33 ± 1.1513.33 ± 3.063.22 ± 0.11
C1: Mixed-type89.33 ± 1.1542.67 ± 1.152.54 ± 0.22
C3: Same-user94.00 ± 2.0044.67 ± 3.062.16 ± 0.09
DSC0: No history68.67 ± 4.1614.00 ± 2.004.45 ± 0.39
C1: Mixed-type74.67 ± 1.1518.00 ± 5.293.89 ± 0.19
C3: Same-user76.67 ± 1.1524.00 ± 4.003.65 ± 0.03

The trend is consistent across all three runs: Same-user history improves FT-ES and reduces TTC over No history for every model. For GPT-5.5, matched history also yields higher ES, higher or tied FT-ES, and lower TTC than shuffled history in every run;

For the regression highlighted in Q2, we additionally tested GPT-5.6-Sol. In the original Table 1, Same-user ES was 78.7% versus 79.0% with No history, a 0.3 pp decrease. We repeated all three history conditions three times, with 50 tasks per condition per run and the same history-sampling protocol. The table reports mean ± sample standard deviation.

HistoryES ↑FT-ES ↑TTC ↓
No history77.33 ± 1.154.00 ± 2.004.44 ± 0.00
Shuffled85.33 ± 1.1520.00 ± 3.463.22 ± 0.21
Same-user90.67 ± 1.1523.33 ± 4.162.86 ± 0.07

GPT-5.6-Sol shows the same trend: Same-user history improves ES, FT-ES, and TTC over No history in all three runs.

Q1: Independent human ratings on generated data quality

We conducted an additional human evaluation covering dialogue quality and recognizable ambiguity types. Six annotators with computing backgrounds independently assessed 48 dialogues sampled from 300 histories. Each dialogue had two annotators, yielding 96 rating sets.

For dialogue quality, annotators rated the realism of the initial request and the recoverability of the intended implementation after reading the full dialogue on a 1–5 scale.

DimensionMean±SD (1–5)Ratings ≥4Share ≥4
Naturalness/ambiguity realism3.60±0.8660/9662.50%
Intent recoverability4.39±0.8085/9688.54%

Most ratings were at least 4 for both dimensions, with stronger support for recoverability. This supports generally realistic requests and recoverable intent in the sampled histories.

To check whether the generated dialogues contain the intended ambiguity types, each annotator independently selected two of the six mechanisms for each dialogue.

MeasureResult
Shared-label agreement97.92% (47/48)
Mean Jaccard56.25%
Chance agreement67.97%
Chance-corrected κ0.935

Annotators agreed on at least one ambiguity type in 47 of 48 dialogues (97.92%), supporting the recognizability of ambiguity types in the generated data. This suggests that the generation pipeline produces recognizable ambiguity as intended.

We sincerely thank you for your valuable feedback. As the rebuttal period is approaching its end, we are glad to provide further clarifications regarding your remaining questions. We hope these responses adequately address your concerns

Reviewer_2GvyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines a new task: given a user's five previously resolved coding sessions, resolve that same user's recurring ambiguity in a fresh session, ideally without asking a clarifying question. It introduces CAPA, a benchmark of 600 sessions in 60 user-by-ambiguity cells with 300 held out, all synthesized from HumanEval through a three-stage LLM pipeline using six ambiguity mechanisms. The claims are that history helps 11 of 12 models on eventual success (ES) and 12 of 12 on first-turn success (FT-ES) (Table 1), that the gains are personalized rather than generic in-context learning (Table 3), and that mem0 and A-mem do not help while a gating trick does (Tables 4 and 5).

Critical strengths
  1. The task framing is clean. ES, FT-ES and TTC separate "eventually right" from "right without asking", and Figure 1 conveys the setup. The blank control (Appendix A.1, Table 6) is the right sanity check: 100% ES on raw HumanEval shows the drop really comes from the injected ambiguity.

  2. The shuffled-history control (Table 3) is exactly the right question to ask, and the authors deserve credit for including it even though it undercuts their framing (see weakness 1). The evaluation covers 12 models including open frontier models, data and code are in the supplement, and the taxonomy received an actual human check (20 participants, Fleiss kappa 0.66, Section 4.1).

Critical weaknesses
  1. The "personalized" effect mostly is not personalization. In Table 3, shuffled history takes GPT-5.5 FT-ES from 2.3 to 29.0, and correctly matched history reaches 31.0; DeepSeek shows 15.7 versus 17.7. Only GLM shows a real gap (34.7 versus 46.7). ES is flat or slightly worse with matched history. So roughly 90% of the gain is generic "here are five solved dialogues" in-context learning. The authors note the control is contaminated (a 35/59 chance the shuffled history shares a mechanism) but still frame the result as personalization.

  2. The user simulator is oracle guided. In Appendix C.4.3 the clarification planner sees the ground truth and tells the simulated user the single most misaligned aspect to clarify next, so every clarifying question receives a targeted hint toward the answer. Under this protocol ES and TTC measure how many oracle hints are needed, not real clarification ability. This is never mentioned in the main text.

Questions for authors

Clarification requests:

  1. Which model generated the data (Conversation LLM, user agent, J_gen)? Is it one of the 12 evaluated models?
  2. Does the held-out evaluation use the same ground-truth-aware clarification planner as data generation? If so, what happens with a planner that does not see the ground truth?
  3. Why is ES with matched history at or below shuffled history for GPT-5.5 (84.3 versus 85.0)? Can you provide seeds and confidence intervals to separate signal from noise?
  4. Is the ambiguous request stripped of the HumanEval function name and signature? Is there any check that models are not simply recalling the original task?
6 comments

Thank you for recognizing the task framing, controls, and evaluation. We will upload a revision with changes highlighted in blue.

W1: Personalization and generic in-context learning

The original Table 3 compares No history (C0), shuffled history (C1), and Same-user history (C3). We agree that shuffled history is not a purely generic baseline: other users can share the target's recurring ambiguity patterns. In our task, personalization means adapting to the target user's own recurring ambiguity and resolution patterns, which persist across that user's sessions (Sec. 3). Other users may share some of these ambiguity types, so shuffled history can contain evidence relevant to the target user. To separate this evidence from generic examples, we added C2: five other-user sessions with no overlap in the target's ambiguity types. This retains the dialogue context but removes target-type overlap. C2−C0 measures the generic context gain, C1−C2 estimates the contribution of target-relevant ambiguity types within shuffled history, and C3−C2 measures the Same-user gain beyond generic examples.

Because the API snapshots changed after submission, we reran all Table 3 conditions and C2 for GPT-5.5 and DeepSeek V4 Pro. All conditions use the same subset of 12 users with five held-out sessions each, giving 60 sessions per model–condition pair. Prompts, history format, model parameters, and the eight-turn protocol remain unchanged. The table below shows this rerun results.

ES is success within eight turns and FT-ES is first-turn success before clarification, both in %. TTC is mean completion turns, assigning eight to unfinished sessions. pp denotes percentage points.

ConditionMetricGPT-5.5DS
C0: No historyES ↑95.075.0
FT-ES ↑18.313.3
TTC ↓3.053.75
C2: No type overlapES ↑90.080.0
FT-ES ↑33.315.0
TTC ↓2.653.63
C1: Mixed-typeES ↑93.381.7
FT-ES ↑43.321.7
TTC ↓2.273.37
C3: Same-userES ↑95.086.7
FT-ES ↑53.323.3
TTC ↓1.953.00
Total history gain: C3−C0ΔFT-ES (pp)+35.0+10.0
Generic gain: C2−C0ΔFT-ES (pp)+15.0+1.7
Relevant-type increment in mixed history: C1−C2ΔFT-ES (pp)+10.0+6.7
Additional Same-user increment: C3−C1ΔFT-ES (pp)+10.0+1.7
Personalized gain over generic examples: C3−C2ΔFT-ES (pp)+20.0+8.3
Personalized share: (C3−C2)/(C3−C0)%57.183.3
  • Context gain. Even without target-type overlap, solved dialogues improve FT-ES (C2−C0) by 15.0 pp for GPT-5.5 and 1.7 pp for DeepSeek.

  • Personalized gain. Same-user history improves FT-ES over the generic control (C3−C2) by 20.0 pp for GPT-5.5 and 8.3 pp for DeepSeek. These gains account for 57.1% and 83.3% of the total history gain, so user-relevant patterns contribute more than generic examples for both models. Shuffled history also exceeds C2 for both models, so its benefit is not purely generic. The smaller absolute gains for DeepSeek reflect its weaker use of history overall: in the main results (Table 1), history raises its FT-ES by only 2.3 pp, versus 28.7 pp for GPT-5.5.

We will rewrite the discussion of Table 3 to report the generic context gain and the personalized gain separately, and update Table 3 with C2.

W2 & Q2: Ground-truth-aware user simulation

Yes, held-out evaluation uses the same GT-aware planner, but only within the user simulator. The evaluated model neither uses the planner nor sees ground truth (GT). Our description did not clearly distinguish these roles, and we will clarify them in the main text.

The pipeline has two stages: data generation (Sec. 4) and model evaluation (Secs. 3 and 5).

Generation uses a Conversation LLM, a user simulator (GT-aware planner, expression agent, and generation judge), and an execution environment. We inject personalized ambiguity by omitting or obscuring task requirements according to each user's fixed ambiguity types and resolution patterns, then phrasing the request in their communication style. The Conversation LLM responds, the environment tests its code, and the planner uses GT to guide the expression agent's next reply. The generation judge, J_gen, is an LLM that uses task information, submitted code, and execution results to decide whether the generated session is complete. The same user patterns are maintained across sessions.

During evaluation, the model under test replaces the Conversation LLM, while the simulator retains its GT-aware planner and expression agent. Unlike J_gen, the evaluation judge J uses only hidden-test execution in the IDE to determine success. The tested model receives history, the held-out request, user replies, and execution feedback, but no GT, planner outputs, or hidden tests. User replies can still provide targeted information, which we examine in the ablation below.

The planner compares the dialogue and execution results with GT to identify which requirements still need clarification. This function depends on GT, so we jointly remove GT and the planner from the user-reply mechanism in the ablation. We compared three settings. A uses the original GT-aware planner to generate user replies. B removes the planner and user replies, leaving only test pass/fail results and error types. Concrete test values are hidden. C adds a persona-conditioned user to B. Both GT and the planner are removed. The user generates replies directly from dialogue and execution feedback and must not invent requirements. We evaluated DeepSeek V4 Pro and GLM-5.2 with No history and Same-user history on the same 20 users, each with five held-out tasks (100 tasks per model–history pair). A/B/C share the first response, so their FT-ES is identical by design. Recovery is the percentage of first-turn failures solved within eight turns.

ModelHistorySettingFT-ES ↑ES ↑TTC@8 ↓Recovery ↑
DSNoA13.074.04.2170.1 (61/87)
B13.027.06.5116.1 (14/87)
C13.025.06.5713.8 (12/87)
SameA20.076.03.7670.0 (56/80)
B20.039.05.7523.8 (19/80)
C20.031.05.9713.8 (11/80)
GLMNoA18.083.03.3779.3 (65/82)
B18.032.06.0917.1 (14/82)
C18.032.06.2317.1 (14/82)
SameA31.080.03.2271.0 (49/69)
B31.050.04.9227.5 (19/69)
C31.039.05.4611.6 (8/69)

We use this protocol to simulate a user who knows their intended requirements. GT provides this information, and the planner selects what to clarify next. Giving the simulator the full specification is common practice in clarification benchmarks: Ambig-SWE's user proxy receives the fully specified issue, HumanEvalComm's evaluator sees the original problem when answering clarifying questions, and Stateful SWE's simulator has access to the complete issue and hints. ES and TTC therefore measure success and interaction cost under this protocol, not unrestricted real-world clarification ability. FT-ES is computed on the first response, before any user reply, so it is not affected by the simulator's replies.

The ablation compares simulating the user with and without GT. Without it, performance drops substantially. Removing the planner and user replies (B) lowers ES by 30–51 pp and recovery from 70.0–79.3% to 16.1–27.5%. Adding a no-GT user (C) does not restore performance; Same-user ES falls further from 39% to 31% for DeepSeek and from 50% to 39% for GLM. We attribute the drop to the language model rather than to the protocol design. We analyzed the interaction logs and found that without GT, the model simulating the user loses track of the intended requirements and cannot stably play the user, so its replies no longer steer the dialogue toward them. GT is therefore what makes the user simulation reliable.

History also helps without targeted replies in the current session. Same-user history raises FT-ES by 7 pp for DeepSeek and 13 pp for GLM before any clarification. Under B/C, it improves ES by 6–18 pp and reduces TTC by 0.60–1.17 turns. The protocol therefore affects absolute performance, but current-session hints do not explain all the history gains.

We will update the paper correspondingly and upload the edited paper.

Q3: Matched versus shuffled history and run-to-run variability

GPT-5.5 achieves 84.3% ES with matched history and 85.0% with shuffled history. The 0.7 pp gap is small, so ES offers limited separation between these conditions. We chose relatively simple HumanEval tasks to separate personalized ambiguity from coding difficulty. Once the requirements are clarified, GPT-5.5 can usually solve the tasks under either history condition. High eventual success in both conditions is therefore consistent with our expectations.

We ran DeepSeek V4 Pro ten times per condition on ten fixed tasks from users T02/T03, keeping histories and parameters fixed. The table reports mean ± sample standard deviation across the ten runs, followed by the approximate 95% t CI for the mean [df=9]. ES and FT-ES are percentages (SD in percentage points); TTC is in turns.

ConditionES ↑FT-ES ↑TTC ↓
No history69.00 ± 12.87 [59.80, 78.20]9.00 ± 3.16 [6.74, 11.26]4.47 ± 0.70 [3.97, 4.97]
Shuffled80.00 ± 9.43 [73.26, 86.74]10.00 ± 0.00 [10.00, 10.00]3.79 ± 0.48 [3.45, 4.13]
Same-user84.00 ± 8.43 [77.97, 90.03]11.00 ± 3.16 [8.74, 13.26]3.44 ± 0.40 [3.16, 3.72]

Same-user TTC has relatively low run-to-run variation (SD 0.40 turns) and a mean CI of [3.16, 3.72], although ES still fluctuates. The zero-width shuffled FT-ES interval reflects identical values in these ten runs, not certainty on new tasks.

We also repeated No history (C0), shuffled history (C1), and Same-user history (C3) three times for GPT-5.5 and DeepSeek V4 Pro with the current API versions and unchanged evaluation settings. This experiment uses a separate subset of 50 held-out sessions from ten users, different from the 60-session subset in W1, so absolute values differ from the W1 table. Shuffled histories are resampled with seeds 0, 1, and 2; the other conditions keep identical inputs across runs. The table reports mean ± sample standard deviation across three runs.

ModelConditionES ↑FT-ES ↑TTC ↓
GPT-5.5C0: No history89.33 ± 1.1513.33 ± 3.063.22 ± 0.11
C1: Mixed-type89.33 ± 1.1542.67 ± 1.152.54 ± 0.22
C3: Same-user94.00 ± 2.0044.67 ± 3.062.16 ± 0.09
DSC0: No history68.67 ± 4.1614.00 ± 2.004.45 ± 0.39
C1: Mixed-type74.67 ± 1.1518.00 ± 5.293.89 ± 0.19
C3: Same-user76.67 ± 1.1524.00 ± 4.003.65 ± 0.03

The trend is consistent across all three runs: Same-user history improves FT-ES and reduces TTC over No history for every model. For GPT-5.5, matched history also yields higher ES, higher or tied FT-ES, and lower TTC than shuffled history in every run; the original ES reversal is not reproduced. Its mean ES difference is +4.67 pp.

Q1: Data-generation model

During data generation, the Conversation LLM, user agent, clarification planner, and generation judge (J_gen) all use GPT-5.5, one of the 12 evaluated models. During evaluation, success is determined by executing the submitted code against hidden tests in the IDE, not by an LLM judge.

Q4: HumanEval identifiers and task recall

Yes, we remove the HumanEval function name and signature from the initial request. The table compares FT-ES on original HumanEval tasks and CAPA requests without history: the large drop shows that task familiarity alone does not resolve the ambiguity.

ModelOriginal FT-ESCAPA, No history FT-ES
GPT-5.589.0%2.3%
DS91.5%15.3%

Comments on the paper

1 comment

Dear Area Chair,

We thank you for your time and effort. Below is a concise summary to assist your decision.

Reasons to publish: CAPA asks a question that standard coding benchmarks never ask: when a returning user keeps omitting the same kind of requirement in the same way (personalized ambiguity), can an assistant learn that habit from their resolved sessions and get the new request right without asking again? Our contribution makes this measurable: each user's ambiguity pattern is fixed across tasks, history is varied with mechanism-aware controls, and models are judged by executable correctness and clarification cost. The reviewers agreed on three points. (1) The question is interesting and practical: 7qZM called it "a useful problem that ordinary coding evaluations largely miss" and said the value of a query log is established "without a doubt"; ho3u found the prior-session setup realistic. (2) The design is sound: 2Gvy praised the "clean" task framing and the shuffled-history control as "exactly the right question to ask"; 7qZM agreed it is a proper baseline. (3) The study is informative: 2Gvy credited the 12-model evaluation and human-checked ambiguity taxonomy; ho3u found "useful insights".

During the discussion, we answered each concern raised by Reviewers with new evidence and revised the manuscript:

  1. Q1 (2Gvy & 7qZM, W1): Is the gain truly personalization, or just generic in-context learning from solved dialogues in table 3? We added a mechanism-disjoint control and reran all four history conditions on the same sessions. First-turn success rises monotonically with user-relevant evidence, and same-user history adds +20.0 / +8.3 / +3.3 pp over a purely generic baseline, so the gain is tied to the user's recurring ambiguity rather than generic examples.

  2. Q2 (2Gvy & 7qZM, Q2/Q3): Are the small ES differences and the GPT-5.6-Sol regression signal or noise? We repeated the main comparisons across three seeds with paired differences and ran a ten-run confidence-interval study. All 27 model-metric-seed comparisons favor same-user history; neither reversal recurs, so both are run-to-run noise.

  3. Q3 (2Gvy, W2/Q2): Does the ground-truth-aware user simulator make ES and TTC measure oracle hints rather than clarification? We compared three user-side settings: the original informed user, execution feedback only, and a user who cannot see ground truth. (i) Nothing is leaked: ground truth and planner outputs stay with the simulated user, unseen by the model, and the same simulator is shared by all conditions. (ii) Without ground truth the protocol does not work: clarifying questions go unanswered, recovery falls from 70-79% to 16-28%, and a no-GT user is worse. (iii) This follows common practice: Ambig-SWE, HumanEvalComm and Stateful SWE all give their simulated user the full specification. FT-ES is computed before any user reply and is identical across settings, and the history gain persists in all three, so hints do not create it.

  4. Q4 (ho3u & 2Gvy, W1/W2/Q1): Why HumanEval rather than a newer or agentic benchmark, is CAPA saturated, and can models recall the original tasks? We want to measure ambiguity, but coding performance reflects both coding ability and ambiguity, so we use HumanEval to minimize the influence of coding ability and isolate ambiguity. Without ambiguity, models are essentially at ceiling (100% ES, about 90% first-turn success); adding ambiguity to the same tasks cuts first-turn success to 2-15%, so the drop comes from ambiguity alone. High ES is by design, not saturation. Extending the same pipeline to BigCodeBench-Hard shows the entanglement: debugging dominates the interaction and coding difficulty cannot be separated from ambiguity. A harder or agentic base therefore cannot answer our question, which is why HumanEval is preferred.

  5. Q5 (7qZM, Q1): Were the generated histories independently checked for realistic ambiguity and recoverable intent? Six independent annotators rated 48 dialogues. Intent recoverability averages 4.39/5 and the mechanisms are recognized in 47/48, so the histories provide valid disambiguation evidence rather than artefacts of the generator.

  6. Q6 (ho3u & 2Gvy, W3-W5/Q1-Q2): What differentiates CAPA?

  • Two lines of work meet here: ambiguity-and-clarification benchmarks (Ambig-SWE, HumanEvalComm, EDIT-Bench) treat ambiguity as a property of the task, so missing requirements must still be asked for; user modeling (ToM-SWE) uses history only for preferences and style. CAPA sits at their intersection: each user's ambiguity pattern stays fixed across tasks, so their resolved histories are disambiguation evidence and models can infer missing requirements without asking. We cited the recommended works, added a comparison table, and narrowed the claim from 'history helps' to a controlled test of recurring user ambiguity.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.