acceptodds
Under review as a conference paper at ICLR 2027

Contra: Discovering and Qualifying Behavior-Changing Questions for Selective Clarification in LLM Code Generation

Abstract

Coding agents can generate code that appears correct but implements behavior the user never intended. This mismatch can arise when an agent silently resolves underspecified requirements through its own assumptions. As subsequent development builds on these assumptions, correcting the resulting behavior can become increasingly costly. Early clarification can help prevent such mismatches, but unnecessary questions can interrupt developers and slow down development. Existing methods struggle to identify key clarification questions while avoiding unnecessary ones. Therefore, we propose CONTRA, a training-free method that combines broad question discovery with semantic and execution-based question qualification. CONTRA first generates candidate questions and filters out those unrelated to required behavior or already resolved by the requirement. For each remaining question, it generates programs conditioned on two plausible answers and checks for stable behavioral differences on shared inputs. It then uses the interaction history to select among qualified questions or stop asking. Experiments on ClarifyCodeBench show that CONTRA achieves the highest F1 with all four coding agents, exceeding the best baseline macro-average F1 by 13.88 percentage points. With the same LLM and evaluation protocol, CONTRA also achieves higher clarification recall and F1 than the coding harnesses Claude Code and OpenHands. To support practical use, we also implement CONTRA as a Claude Code plugin that integrates selective clarification into everyday development.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Selective Clarification, Code Generation, Coding Agents, Requirement Underspecification, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VDV9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CONTRA consists of a training-free framework for selective clarification divided into 3 stages: (1) question discovery (2) question qualification based on filtering and execution (3) adaptive question selection which decides when to ask questions or stop. Their experiments demonstrate consistent improvements across agents and the approach is also presented as a Claude Code plug-in for usability. The Flash variant also aims to reduce cost for qualification. The approach is then also evaluated on real repository issues to demonstrate its real-world utility.

Critical strengths

S1: The evaluation is thorough and the consistent empirical gains demonstrate the usefulness of the method. Overall I find the work very well structured with the choice of benchmark and complimenting it with the repository-level study, and comparisons across harnesses.

S2: The pipeline is very intuitive and well ablated with the Flash model also incorporating the efficiency angle into the pipeline.

S3: The question qualification approach is simple and effective with impact observed for different questions.

Critical weaknesses

W1: The method's novelty is limited as the pipeline combines or adapts multiple components that exist in literature (like relevance and redundancy in Asking What Matters and behavioral disagreements like in ClarifyGPT)

W2: The pipeline is constrained by questions generated in the initial phase. A useful ablation could be tracing the source of final clarification questions after qualifications and their origin (model generated, with context-aware or without). Comparing across models can also show how model-dependent this step is. Finally comparison with human-generated questions for at least some issues can help identify: (1) are the "ideal"/"good" questions identified by the pipeline, (2) what type of generation configuration gets the most productive candidates, (3) how does this vary as the model backbone gets weaker or stronger.

W3: I believe non-functional requirements would not manifest as observable differences in question qualification. Further, the work only focuses on single-function tasks. For other cases, it would be more challenging to observe behavior changes or test for them.

W4: The real repository evaluation is limited and not analyzed in the sufficient depth

Questions for authors

Q1: See W2

Q2: How does the pipeline work for non-functional requirements?

Q3: Could uncertainty estimates and more analysis be included for Table 4?

Q4: How does adaptive selection resolve dependency between questions (the answer to a question changes the content of another question)?

6 comments

Thank you for highlighting Question Qualification as "simple and effective" and the overall pipeline as "very intuitive and well ablated." We appreciate your recognition of how the repository-level study and comparisons across coding harnesses complement the benchmark evaluation. We are especially encouraged by your assessment that "the consistent empirical gains demonstrate the usefulness of the method."

We will incorporate the additional results and discussion below into the revised paper.

W1. What does CONTRA contribute beyond combining ideas from prior work?

CONTRA goes beyond simply combining existing methods through an overall design in which the agent first discovers questions broadly and then uses behavioral evidence to decide which ones to ask. Question Discovery builds a broad candidate set to recover key questions that might otherwise be missed. Question Qualification improves precision by retaining relevant, unresolved questions whose alternative answers produce stable differences in program behavior. After each user response, Adaptive Question Selection uses the updated interaction history to choose a question that still needs an answer or to stop when no further input is needed. This design keeps developers in control of consequential implementation choices without asking them to review every question the agent considers.

Asking What Matters supports the principle that questions should be relevant and non-redundant. It uses these properties to train a clarification policy and weights relevance by information category. CONTRA shares the principle but uses a different, training-free test. For each candidate, it checks whether two plausible answers specify distinct choices about required behavior and whether the requirement already settles that choice. The same two answers then define the execution test.

ClarifyGPT uses behavioral disagreement to generate a question. It samples programs from the same underspecified requirement and looks for differences in their outputs. CONTRA starts with a specific question, generates programs under each of its two plausible answers, and tests for a stable difference on shared inputs. This ties the observed difference to the question being considered. The interaction history then determines whether that question still needs an answer or whether the agent should stop.

We will revise Section 2 to explain this design rationale and distinguish it more clearly from both methods.

Q1 & W2. How well does Question Discovery recover gold questions, and how does each strategy's contribution vary across models?

We traced each final question back to its source in Question Discovery. Independent generation uses only the requirement. Context-aware generation also uses the candidates found so far. We report the source shares separately for final questions that matched a gold question and those that did not. The context-aware and independent columns include shared questions. The last two columns exclude them.

ModelCONTRA F1Gold matchContext-awareIndependentContext-aware onlyIndependent only
GPT-5-mini35.36Yes90.0%95.0%5.0%10.0%
GPT-5-mini35.36No85.4%88.5%11.5%14.6%
Grok 4.536.54Yes90.5%97.6%2.4%9.5%
Grok 4.536.54No90.3%79.1%20.9%9.7%
GPT-5.542.39Yes72.9%98.2%1.8%27.1%
GPT-5.542.39No71.4%88.7%11.3%28.6%
Gemini 3.1 Pro50.49Yes51.2%96.6%3.4%48.8%
Gemini 3.1 Pro50.49No54.2%80.3%19.7%45.8%

Independent generation contributed more unique gold matches on all four models. Its share of matched questions found by no other source ranged from 9.5% on Grok 4.5 to 48.8% on Gemini 3.1 Pro. Independent generation also produced many questions that did not match a gold question. Context-aware generation contributed a smaller but consistent share. On every model, it found some matched questions that independent generation missed. Together with Figure 2 in our paper, which shows that Question Discovery recovers most gold questions within about four rounds, these results support using both sources to build a high-recall candidate set.

Q2 & W3. How does CONTRA handle non-functional requirements?

Thank you for raising this question. We see this as a promising direction for future work. For example, we can run both implementations under the same workload and measure latency or peak memory across repeated runs. This would let us check whether alternative answers lead to a consistent difference in the metric that matters for the requirement.

Q3. Could uncertainty estimates and more analysis be included for Table 4?

We evaluated downstream code generation with GPT-5.5 over 3 runs. CONTRA ranked first and achieved the highest mean pass@1. The table reports the mean and standard deviation across runs.

MethodMean pass@1 ± SD
Direct Prompting63.11 ± 1.61
Ask-or-Assume62.70 ± 1.07
ClarifyGPT64.63 ± 1.33
CONTRA65.04 ± 1.86

Q4. How does adaptive selection resolve dependency between questions (the answer to a question changes the content of another question)?

Thank you for raising this question. The current selector can skip a question that an earlier answer has resolved. Revising the content of a follow-up question based on that answer is a direction for future work.

W4. The real repository evaluation is limited and not analyzed in the sufficient depth.

In repository development, the extra cost of clarification can be worthwhile when it resolves an implementation choice before later changes depend on it or unintended behavior reaches users. Our general response illustrates this point through an image proxy case study. It shows how one question surfaced a privacy concern before deployment and compares the cost of asking upfront with the cost of repairing the implementation after further development.

Overall

Thank you again for your review. We hope the additional experiments and discussion address your remaining concerns. We would be grateful for your support.

Thank you for your response. I especially find the results for (Q1/W2) interesting, though I would prefer more qualitative analysis in the type/diversity of questions and, if possible, comparison with human generated questions in future revision.

However, my other concerns remain largely unresolved: (1) I still find the novelty (W1) to be limited, (2) the E2E gains are not clear (Q3, W4), and (3) the framework has limited support for adaptive selection/non-functional requirements (Q2, Q4 though I find this to be a relatively smaller concern than the others). I do agree the problem of clarification is important, but the work does not convince me of its effectiveness over existing solutions. I will maintain my current score.

Reviewer_J8HsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes CONTRA, a training-free method for selective clarification in LLM-based code generation. It combines iterative candidate-question discovery, semantic filtering for relevance and unresolved requirements, and execution-based verification that checks whether two plausible answers produce stable behavioral differences across generated programs. An interaction-history-aware selector then chooses which qualified question to ask or when to stop.

Critical strengths
  1. The method introduces a concrete, execution-grounded criterion for qualifying clarification questions. Rather than relying solely on an LLM’s judgment about whether clarification is needed, CONTRA generates programs under two alternative answers and checks for stable behavioral differences on shared inputs. This directly connects a candidate question to its potential implementation consequences and provides a meaningful methodological distinction from detecting disagreement among programs sampled from the same underspecified requirement.
Critical weaknesses
  1. The central empirical claims rely on an outdated evaluator without validating its reliability. Section 4.1 states that GPT-4o matches generated clarification questions against gold questions, with majority voting over three judgments. These decisions directly determine precision, recall, F1, and TKQR, which underpin the paper’s principal performance claims. Yet GPT-4o(released in Oct. 2024) is used to evaluate questions generated by substantially newer models, including GPT-5.5 and Grok 4.5, without demonstrating that it can reliably assess their semantic equivalence to the gold questions. Notably, OpenAI retired GPT-4o from ChatGPT on February 13, 2026. Protocol consistency is not a substitute for evaluator validity. The authors should cross-check the results using a strong contemporary evaluator with demonstrated reliability on comparable tasks and validate a representative subset through human annotation. Majority voting over the same model does not address systematic matching errors. Without these checks, the magnitude of the reported gains and the stability of the method rankings remain insufficiently substantiated, weakening the empirical basis for the paper’s main contribution.

  2. The paper does not establish that reduced questioning justifies the additional latency and computational cost. The motivation is to reduce unnecessary interruptions to developers, but the evaluation emphasizes question precision and coverage without establishing the trade-off between user effort and system overall time consumption. Appendix C illustrates this concern: coding takes 40 seconds and costs $0.47, while reporting takes another 90 seconds ($0.17), with an additional 15 seconds ($0.06) of background verification; the report becomes available approximately 145 seconds after the request (2.6x more time!). Thus, in this example, obtaining the report requires substantially more waiting than completing the coding task itself. However, the paper does not compare this overhead against simply asking additional questions, potentially in a single batch, nor does it report representative end-to-end latency, token usage, or monetary cost for the main CONTRA pipeline. The proposed workflow may not be preferable in practice. This missing validation of the paper’s central practical premise is a major reason for my negative assessment.

Questions for authors
  1. How reliable is GPT-4o as a question-matching evaluator in this setting?
  2. What are the actual end-to-end costs(time, money, tokens, etc) of the main CONTRA pipeline?
  3. Does selective filtering outperform simply asking more questions under a comparable total time budget?
10 comments

Thank you for highlighting CONTRA's "concrete, execution-grounded criterion for qualifying clarification questions." We appreciate your observation that testing two plausible answers "directly connects a candidate question to its potential implementation consequences." As you emphasize, this link distinguishes CONTRA from methods that detect disagreement among programs sampled from the same underspecified requirement.

We will incorporate the additional results and discussion below into the revised paper.

W1 & Q1. How reliable is GPT-4o as a question-matching evaluator in this setting?

We used GPT-4o to match generated questions against gold questions, following the ClarifyCodeBench evaluation protocol.

To check the reliability of these judgments, we repeated the evaluation with GPT-6-astra for all four methods across all four generating models. We also manually reviewed 50 sampled judgments. GPT-4o and GPT-6-astra agreed on 90.81% of the 6,591 paired judgments. GPT-4o also agreed with the human judgment in 46 of the 50 sampled cases, an agreement rate of 92%.

The table reports F1 and TKQR under GPT-6-astra, both as percentages.

Generating modelDirect PromptingAsk-or-AssumeClarifyGPTCONTRA
GPT-5-mini14.05 / 12.7216.20 / 16.3318.42 / 29.3134.03 / 33.61
Grok 4.522.50 / 17.5023.13 / 21.7519.53 / 26.8339.14 / 39.45
Gemini 3.1 Pro38.64 / 35.3038.28 / 36.0518.81 / 19.9651.72 / 47.80
GPT-5.522.03 / 15.4821.98 / 20.0520.27 / 25.8644.64 / 39.95
Macro-average24.30 / 20.2524.90 / 23.5519.26 / 25.4942.38 / 40.20

With GPT-6-astra as the evaluator, CONTRA still achieves the highest F1 and TKQR on all four generating models. Its macro-average F1 and TKQR exceed the best baseline for each metric by 17.5 and 14.7 percentage points, respectively.

W2 & Q2. What are the actual end-to-end costs(time, money, tokens, etc) of the main CONTRA pipeline?

Token usage

Here is the token usage on ClarifyCodeBench. Here we compare with the baseline ClarifyGPT.

ModelCONTRAClarifyGPT
Gemini 3.1 Pro47.8k77.5k
GPT-5-mini89.0k60.7k
GPT-5.575.8k54.9k
Grok 4.5213.5k195.4k

CONTRA does not always use more tokens than ClarifyGPT. Its token usage depends in part on how many candidates the model generates during Question Discovery. A larger candidate set requires more tokens for Question Qualification, while a smaller set costs less.

When CONTRA does use more tokens, it can achieve better clarification results than ClarifyGPT. On GPT-5-mini, CONTRA achieved 35.36% F1 and 34.32% TKQR, compared with 19.92% and 30.94% for ClarifyGPT. The image proxy case in our general response illustrates why identifying a key question can be worth the extra cost. Asking whether visitors could disable the proxy resolved a privacy issue before deployment and avoided a more expensive repair later.

Time to the initial report

In Appendix C, coding took 40 seconds, and the earlier plugin needed another 90 seconds to display its report. We updated the plugin. The updated plugin displays an initial report as soon as coding finishes. The developer can give the next instruction immediately. Further checks, including Behavioral Verification, run in the background. The plugin updates the report when the results are ready.

The initial report highlights implementation choices that Claude Code rarely makes explicit. In Table 6, only 4.3% of the decisions reported by Claude Code stated what the requirement left unspecified, compared with 59.7% with CONTRA. Behavioral Verification then provides execution evidence. A distinguishing test shows how the current implementation and an alternative behave differently on the same input. If the developer prefers the alternative, the plugin can reuse the tested patch.

Case study

In our general response, we provide a case study from the repository evaluation. It illustrates why identifying a key question can be worth the extra cost. Asking whether visitors could disable the proxy resolved a privacy issue before deployment and avoided a more expensive repair later.

Q3. Why filter candidate questions rather than simply ask more?

Figure 3 in our paper shows why generating more questions is not enough. Question Discovery recovers most gold questions, yet its F1 remains low. This means the candidate set also contains many unnecessary questions. The low macro-average TKQR of 19.03% indicates that key questions often appear late in discovery order. Even when the model has found a key question, the developer may have to read through many unnecessary questions before reaching it. Asking all candidates in one batch would still leave that screening to the developer.

CONTRA filters candidates before they reach the developer. Across the four models, it filtered out 94.7% of the candidates on average.

ModelGPT-5-miniGPT-5.5Grok 4.5Gemini 3.1 ProMacro-average
Candidates filtered out96.9%98.7%99.0%84.3%94.7%

Filtering also brings key questions forward. Question Qualification alone raises macro-average TKQR from 19.03% to 32.52% without changing the order of the remaining candidates. The gain comes from removing questions that would otherwise appear ahead of key ones. This lets developers focus on the decisions that need their input.

Overall

Thank you again for your review. We hope the additional experiments and discussion address your remaining concerns. We would be grateful for your support.

  1. Actual computational overhead.

Could the authors add a clean baseline (i.e., the original coding agent without any clarification mechanism) to the token usage table? Besides, two additional columns for end-to-end time consuption would be better. This would help quantify the actual additional token consumption introduced by CONTRA, rather than only comparing it against ClarifyGPT.

I'm concerning the end-to-end time consuption. Not a comparison to other baselines. I never mention clarifygpt in weakness2.

Performance-Latency tradeoff shouldnot be ignored. Comparing against ClarifyGPT, a 2024 method, is insufficient to establish the practical advantage of CONTRA with modern coding agents.

  1. 95% filter rate?

In my experience with LLM-based coding agents, it is uncommon for an agent to raise more than a handful of clarification questions in a single interaction. Does this high filtering rate mean that the discovery stage generates an excessively large number of candidate questions, most of which are subsequently discarded?

  1. "case study"

The authors should not rely on a single carefully selected case study to demonstrate the practical benefits of CONTRA. If you wish to report the time and cost savings, they should evaluate multiple repository-level tasks and report the mean and standard deviation. Presenting only one example is counterproductive, as it provides little statistical evidence and raises concerns about cherry-picking.

Reviewer_UGjpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors proposed a training-free workflow for coding agents to ask the right clarification questions at the right time. The problem to be solved is, coding agents tend to either (1) ask for clarifications frequently, which hinders the development progress, or (2) not asking clarification questions enough, instead, acting aggressively by making strong assumptions silently, leaving users unaware. The former is bothering while the latter can be destructive, because it can be very costly to recover a codebase from such deviated assumptions, especially when a project proceeds and accumulates mental gaps over time. The problem is real and practical.

The authors borrow classic ideas (e.g., from recommendation systems & information retrieval) to (1) gather clarification question candidates broadly, keeping a high recall rate, then (2) filtering such candidates rigorously to increase precision, and finally (3) adaptively choosing the right clarification questions over time while updating the interaction history dynamically, preventing asking stale questions that had been solved. Experiments are conducted on multiple different agents to show robustness against agent selection; ablations are also performed against popular harnesses.

Critical strengths

S1 (Novel formulation and practical approach). The main strength of this paper is its contrastive behavioral mechanism, which converts the clarification & code revision problem (sometimes can be quite ambiguous) to an objective and verifiable problem (by measuring the actual behavioral differences of code). This echoes with another idea of code-as-world-model, where in certain scenarios, using code's execution space to replace that of natural language description and modeling can be advantageous. To me this is analogous to converting semantic near-dedup to verifiable near-dedup; it reliably removes redundant clarification questions.

S2 (Technical Rigor). Regarding this contrastive behavioral mechanism, there are two natural questions to raise; the paper has successfully addressed them.

Question 1: is it possible if the same question can result in different implementations due to interpretation variance and the stochasticity of sampling? any experiment to back up this?

This question has been addressed in section 3.4 with rigor: multi-sample generation and majority voting are used to reduce sampling noise. Ablation experiment in section 4.6 also shows the effect of removing the behavioral verification.

Question 2: assuming two questions are indeed different and non-redundant after filtering with this contrastive behavioral mechanism, what is the main benefit from it? The paper explicitly explains that this reduces the developer's disturbance cost; in solid metrics, it achieves lower FPR and higher efficiency (higher TKQR). This echoes with what problem the paper earlier claimed to tackle.

Critical weaknesses

W1. Confounding factors and inadequate ablations (Major).

What the paper has: different models (gpt-5.5, gpt-5-mini, grok4.5) x workflow set (contra, direct prompting, ask-or-assume, clarifyGPT)

What the paper doesn't have yet: different models (gpt-5.5, gpt-5-mini, grok4.5) x oss harnesses vs different models x oss harnesses + this plugin

Details below:

when evaluating frontier models with Contra on ClarifyCodeBench, GPT-5 mini, GPT-5.5, Grok4.5, Gemini3.1-Pro are used (to show Contra brings consistent merit to different models); however, when comparing CONTRA with harnesses like ClaudeCode and OpenHands, Qwen3.8-27B is the only model being used. This leaves a few questions:

Q1. Qwen3.8-27B does not achieve frontier-level performance, will this ablation provide conclusions that generalize to frontier models? We know that for weaker and stronger models, their sensitivity to harnesses can vary significantly.

To prove that the advantage of CONTRA over other harnesses' default clarification mechanism does not fade away for stronger models of larger scale, I suggest adding at least one frontier model (if cost is high) to ablate with, comparing CONTRA with other harnesses.

Q2. Does CONTRA suits models better than other harnesses consistently? Are there cases where certain models utilize certain frameworks/harnesses better (regardless of the capability gap)? Is there any implicit bias here that Qwen3.8-27B just adapts to CONTRA the best?

To prove the cross-model robustness, the same suggestion is to add at least one frontier model to perform the same ablation experiments.

Q3. Is this task-specific or consistent across different coding benchmarks? Can this be generalized to similar clarification-related benchmarks? Is there any bias in this benchmark (such as metrics and eval logic) to favor certain frameworks such as CONTRA?

If time allows, adding another benchmark to ablate would make it much stronger. Otherwise, I encourage authors to add some text to discuss the application boundaries and potential limitations of CONTRA on real-world scenarios/benchmarks beyond ClarifyCodeBench.

W2. Cost Analysis (Minor). The tradeoff of using CONTRA seems to be its additional computational overhead. Q4. Does this bring extra token cost & latency significantly or not? It might be worthwhile if there is some analysis of this. This is a practical paper and CONTRA is said to be provided as a Claude Code plugin, so the token cost and latency becomes realistic concern for readers and users.

W3. Sensitivity Analysis (Minor). In section 3.4, the hyper-params (3 programs, 7 inputs) seem to be pre-determined. Q5. Are they chosen randomly or are they chosen to achieve the optimal performance based on ablation experiments? To demonstrate the robustness, I suggest the addition of a small ablation study for sensitivity analysis if time allows; otherwise, I encourage authors to describe what has been tried, what works or does not work, and why.

Questions for authors

My questions (Q1-Q5) whose answers would affect the assessment have been mentioned in the weaknesses section above, closer to the context around which they were raised.

Additional feedback on potential limitations:

Does this CONTRA framework apply well to broader coding scenarios where in principle, the core assumption that the output of the code is deterministic does not suffice (e.g., stochastic algorithms, concurrency, etc)? Adding some text to describe the potential limitation might be beneficial.

3 comments

We appreciate your recognition of the value of discovering candidate questions broadly and then filtering them rigorously. We are especially encouraged that you identify the contrastive behavioral mechanism as the paper's "main strength" and find that it "reliably removes redundant clarification questions." This is the balance we aimed for in CONTRA: finding key questions without burdening developers with unnecessary ones.

We will incorporate the additional results and discussion below into the revised paper.

Q1 & Q2. Do CONTRA's gains over other coding harnesses extend to frontier models?

We compared CONTRA, Codex, and Claude Code on ClarifyCodeBench. All three used GPT-5.6-sol. CONTRA outperformed both coding harnesses on F1 and TKQR.

MethodF1 ↑TKQR ↑Precision ↑Recall ↑FPR ↓
Codex29.1921.1962.8619.010.00
Claude Code26.7119.5258.8217.280.00
CONTRA37.6632.4944.5432.611.25

CONTRA exceeded the stronger baseline, Codex, by 8.47 percentage points in F1 and 11.30 percentage points in TKQR. The higher F1 reflects a better balance between asking unnecessary questions and missing key ones. The higher TKQR indicates that CONTRA identified key questions earlier and used clarification turns more efficiently. These gains in selective clarification extend beyond Qwen3.8-27B to the frontier model GPT-5.6-sol.

Q3. Generalization beyond ClarifyCodeBench

CONTRA can also identify unresolved implementation choices in real repository tasks. In the image proxy case described in our general response, CONTRA surfaced a privacy decision that Claude Code had left implicit. This shows that CONTRA can identify consequential choices outside ClarifyCodeBench. We plan to evaluate whether its gains in F1 and TKQR also hold on other clarification benchmarks.

Q4. Cost analysis

Here is the token usage on ClarifyCodeBench. Here we compare with the baseline ClarifyGPT.

ModelCONTRAClarifyGPT
Gemini 3.1 Pro47.8k77.5k
GPT-5-mini89.0k60.7k
GPT-5.575.8k54.9k
Grok 4.5213.5k195.4k

CONTRA does not always use more tokens than ClarifyGPT. Its token usage depends in part on how many candidates the model generates during Question Discovery. A larger candidate set requires more tokens for Question Qualification, while a smaller set costs less.

When CONTRA does use more tokens, it can achieve better clarification results than ClarifyGPT. For example, on GPT-5-mini, CONTRA achieved 35.36% F1 and 34.32% TKQR, compared with 19.92% and 30.94% for ClarifyGPT. The image proxy case in our general response illustrates why identifying a key question can be worth the extra cost. Asking whether visitors could disable the proxy resolved a privacy issue before deployment and avoided a more expensive repair later.

Q5. Hyper-parameter sensitivity

We varied one hyper-parameter at a time using GPT-5.6-sol. The default setting uses 3 programs per completion for majority voting and at least 7 shared test inputs for Behavioral Verification.

Number of programs per completion. We kept all test inputs and varied the number of programs.

Programs per completionF1 ↑TKQR ↑Precision ↑Recall ↑FPR ↓
136.9732.4243.4432.182.50
3 (default)37.6632.4944.5432.611.25
536.8832.1043.1932.181.25

Number of shared test inputs. We kept three programs per completion and varied the number of test inputs. The settings labeled 7 and 9 use at least that many inputs.

Test inputsF1 ↑TKQR ↑Precision ↑Recall ↑FPR ↓
337.4231.6045.1231.970.00
537.8932.5745.2132.610.00
7 (default)37.6632.4944.5432.611.25
937.4132.4344.2532.401.25

Across all tested settings, both F1 and TKQR stayed close to their default values. This indicates that CONTRA's clarification performance is not sensitive to either parameter over the tested range.

Applicability to Stochastic and Concurrent Programs

Thank you for this helpful suggestion. In stochastic or concurrent programs, the same implementation can produce different outputs across runs. How to determine whether a difference is due to the choice being clarified or to randomness or scheduling remains a question for future work.

Overall

Thank you again for your thoughtful review. We hope the additional experiments and discussion address your remaining concerns. We would be grateful for your continued support.

The response does not fully resolve my concerns: (1) it evades the direct discussion on the potential bias in ClarifyCodeBench; (2) regarding token budgets and latency, it is supposed to be a clean ablation on "CONTRA vs no CONTRA"; (3) I hoped to see statistical analysis of the latency, which could have been more meaningful than presenting a single case — especially for potential users of the plugin, a single case is not persuasive in asking them to pay twice the time and token cost.

Therefore, I am not raising the score. Overall, I find strengths in this paper, especially in trying to solve a very practical problem. Because this is a pragmatic contribution, aspects and details of this work should be more solid and handled carefully, with enough rigorous ablation comparisons (careful engineering might bring more alpha in finding the sweet spot between F1 gains and token/latency cost). I find some parts still vague, which might have weakened the persuasiveness of the conclusions.

(One more suggestion, not related to scoring: I’d encourage authors to revise PDF directly to reflect discussions, rather than promising changes in a future version.)

Thank you for your follow-up. We evaluate CONTRA and Claude Code on 24 repository-level tasks. All three settings use Claude Code with gpt-5.6-sol:

  • Clean Claude Code: The original Claude Code agent without any added clarification mechanism.
  • Claude Code + CONTRA: Claude Code with CONTRA enabled.
  • Claude Code + repair: Claude Code revises its implementation based on a bug report.
SettingTasks resolvedResolution rateTotal costMean cost per taskTotal end-to-end timeMean end-to-end time per task
Clean Claude Code8/2433.3%$21.92$0.91209 min8.71 min
Claude Code + CONTRA21/2487.5%$31.41$1.31226 min9.42 min
Claude Code + repair21/2487.5%$29.12$1.21296 min12.33 min

Compared with clean Claude Code, CONTRA required an additional $0.40 and 43 seconds per task on average. With that overhead, it resolved 21 tasks rather than 8, leaving 13 fewer tasks unresolved at the end of initial development.

Compared with repair, CONTRA achieved the same resolution rate of 87.5% while costing $2.29 more and taking 70 fewer minutes across all 24 tasks. The key distinction is when the tasks were resolved: CONTRA reached this rate during initial development, without a separate repair step prompted by a bug report.

We agree with your point about the importance of careful engineering. As the saying goes, "practice makes perfect." We will continue to refine the CONTRA plugin through real-world use.

Overall

Thank you again for your support and encouragement. We hope this response has addressed your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Why Clarification Is Worthwhile

Dear Reviewers, ACs, SACs, and PCs,

We want to emphasize that clarification can be worthwhile, and we will illustrate this point with a case study from our repository evaluation. We will revise our paper accordingly.

  1. Delayed fixes can cost more. When a requirement is underspecified, an agent may choose an implementation without consulting the developer. If that choice does not match the developer's intent, correcting it later may also require revising code built on it.

  2. Unintended behavior can reach users. If an agent leaves an implementation choice unreported, the developer may not notice the mismatch until after release. By then, the consequences can extend beyond the cost of fixing the code. CONTRA flags unresolved choices so developers can check them against their intent before deployment.

  3. The extra cost of early clarification can be worthwhile. CONTRA may use more tokens during clarification than other methods. That investment can be worthwhile when it helps developers correct a mistaken assumption before later code depends on it or unintended behavior reaches users.

Clarification can be especially valuable in real-world development.

Case study

Here is an example from our repository evaluation. (We will attach the case in the revised paper.) The issue asked for the image proxy to be enabled on public instances to protect visitors' privacy. Given this requirement, Claude Code enabled the proxy by default but allowed visitors to turn it off. This was precisely the problem the maintainers discussed. A visitor could disable the proxy without realizing that their browser would then contact image hosts directly. After further development had built on the initial implementation, Claude Code had to revise it so that the proxy stayed on for all visitors.

With the CONTRA plugin, Claude Code raised the unresolved choice during implementation. It asked whether the proxy should be enabled by default or whether visitors should be prevented from turning it off. The developer confirmed that it must stay on. The resulting implementation matched the upstream fix and needed no later repair.

Execution time and cost on this case

StageClaude CodeClaude Code with CONTRA
Initial implementation107 seconds, $0.24271 seconds, $0.57
Later repair544 seconds, $0.56No repair needed
Total651 seconds, $0.80271 seconds, $0.57

Clarifying the requirement added 164 seconds and 0.33 dollars upfront but avoided a repair that took 544 seconds and cost 0.56 dollars. They do not even include the potential cost to visitors if the original implementation had gone live and exposed their IP addresses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.