- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CONTRA consists of a training-free framework for selective clarification divided into 3 stages: (1) question discovery (2) question qualification based on filtering and execution (3) adaptive question selection which decides when to ask questions or stop. Their experiments demonstrate consistent improvements across agents and the approach is also presented as a Claude Code plug-in for usability. The Flash variant also aims to reduce cost for qualification. The approach is then also evaluated on real repository issues to demonstrate its real-world utility.
Critical strengths
S1: The evaluation is thorough and the consistent empirical gains demonstrate the usefulness of the method. Overall I find the work very well structured with the choice of benchmark and complimenting it with the repository-level study, and comparisons across harnesses.
S2: The pipeline is very intuitive and well ablated with the Flash model also incorporating the efficiency angle into the pipeline.
S3: The question qualification approach is simple and effective with impact observed for different questions.
Critical weaknesses
W1: The method's novelty is limited as the pipeline combines or adapts multiple components that exist in literature (like relevance and redundancy in Asking What Matters and behavioral disagreements like in ClarifyGPT)
W2: The pipeline is constrained by questions generated in the initial phase. A useful ablation could be tracing the source of final clarification questions after qualifications and their origin (model generated, with context-aware or without). Comparing across models can also show how model-dependent this step is. Finally comparison with human-generated questions for at least some issues can help identify: (1) are the "ideal"/"good" questions identified by the pipeline, (2) what type of generation configuration gets the most productive candidates, (3) how does this vary as the model backbone gets weaker or stronger.
W3: I believe non-functional requirements would not manifest as observable differences in question qualification. Further, the work only focuses on single-function tasks. For other cases, it would be more challenging to observe behavior changes or test for them.
W4: The real repository evaluation is limited and not analyzed in the sufficient depth
Questions for authors
Q1: See W2
Q2: How does the pipeline work for non-functional requirements?
Q3: Could uncertainty estimates and more analysis be included for Table 4?
Q4: How does adaptive selection resolve dependency between questions (the answer to a question changes the content of another question)?
6 comments
Q3. Could uncertainty estimates and more analysis be included for Table 4?
We evaluated downstream code generation with GPT-5.5 over 3 runs. CONTRA ranked first and achieved the highest mean pass@1. The table reports the mean and standard deviation across runs.
| Method | Mean pass@1 ± SD |
|---|---|
| Direct Prompting | 63.11 ± 1.61 |
| Ask-or-Assume | 62.70 ± 1.07 |
| ClarifyGPT | 64.63 ± 1.33 |
| CONTRA | 65.04 ± 1.86 |
Q4. How does adaptive selection resolve dependency between questions (the answer to a question changes the content of another question)?
Thank you for raising this question. The current selector can skip a question that an earlier answer has resolved. Revising the content of a follow-up question based on that answer is a direction for future work.
W4. The real repository evaluation is limited and not analyzed in the sufficient depth.
In repository development, the extra cost of clarification can be worthwhile when it resolves an implementation choice before later changes depend on it or unintended behavior reaches users. Our general response illustrates this point through an image proxy case study. It shows how one question surfaced a privacy concern before deployment and compares the cost of asking upfront with the cost of repairing the implementation after further development.
Overall
Thank you again for your review. We hope the additional experiments and discussion address your remaining concerns. We would be grateful for your support.
Thank you for your response. I especially find the results for (Q1/W2) interesting, though I would prefer more qualitative analysis in the type/diversity of questions and, if possible, comparison with human generated questions in future revision.
However, my other concerns remain largely unresolved: (1) I still find the novelty (W1) to be limited, (2) the E2E gains are not clear (Q3, W4), and (3) the framework has limited support for adaptive selection/non-functional requirements (Q2, Q4 though I find this to be a relatively smaller concern than the others). I do agree the problem of clarification is important, but the work does not convince me of its effectiveness over existing solutions. I will maintain my current score.
Thank you for highlighting Question Qualification as "simple and effective" and the overall pipeline as "very intuitive and well ablated." We appreciate your recognition of how the repository-level study and comparisons across coding harnesses complement the benchmark evaluation. We are especially encouraged by your assessment that "the consistent empirical gains demonstrate the usefulness of the method."
We will incorporate the additional results and discussion below into the revised paper.
W1. What does CONTRA contribute beyond combining ideas from prior work?
CONTRA goes beyond simply combining existing methods through an overall design in which the agent first discovers questions broadly and then uses behavioral evidence to decide which ones to ask. Question Discovery builds a broad candidate set to recover key questions that might otherwise be missed. Question Qualification improves precision by retaining relevant, unresolved questions whose alternative answers produce stable differences in program behavior. After each user response, Adaptive Question Selection uses the updated interaction history to choose a question that still needs an answer or to stop when no further input is needed. This design keeps developers in control of consequential implementation choices without asking them to review every question the agent considers.
Asking What Matters supports the principle that questions should be relevant and non-redundant. It uses these properties to train a clarification policy and weights relevance by information category. CONTRA shares the principle but uses a different, training-free test. For each candidate, it checks whether two plausible answers specify distinct choices about required behavior and whether the requirement already settles that choice. The same two answers then define the execution test.
ClarifyGPT uses behavioral disagreement to generate a question. It samples programs from the same underspecified requirement and looks for differences in their outputs. CONTRA starts with a specific question, generates programs under each of its two plausible answers, and tests for a stable difference on shared inputs. This ties the observed difference to the question being considered. The interaction history then determines whether that question still needs an answer or whether the agent should stop.
We will revise Section 2 to explain this design rationale and distinguish it more clearly from both methods.
Q1 & W2. How well does Question Discovery recover gold questions, and how does each strategy's contribution vary across models?
We traced each final question back to its source in Question Discovery. Independent generation uses only the requirement. Context-aware generation also uses the candidates found so far. We report the source shares separately for final questions that matched a gold question and those that did not. The context-aware and independent columns include shared questions. The last two columns exclude them.
Independent generation contributed more unique gold matches on all four models. Its share of matched questions found by no other source ranged from 9.5% on Grok 4.5 to 48.8% on Gemini 3.1 Pro. Independent generation also produced many questions that did not match a gold question. Context-aware generation contributed a smaller but consistent share. On every model, it found some matched questions that independent generation missed. Together with Figure 2 in our paper, which shows that Question Discovery recovers most gold questions within about four rounds, these results support using both sources to build a high-recall candidate set.
Q2 & W3. How does CONTRA handle non-functional requirements?
Thank you for raising this question. We see this as a promising direction for future work. For example, we can run both implementations under the same workload and measure latency or peak memory across repeated runs. This would let us check whether alternative answers lead to a consistent difference in the metric that matters for the requirement.