- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how false statements in task descriptions cause code failures. A model may solve a task correctly but fail when a misleading statement is added. PSR uses text features to predict this risk before code generation. It samples tasks and uses an extra prompt to check false assumptions. The experiments compare this selection rule with task difficulty.
Critical strengths
- The study separates task difficulty from the risk of following a false statement. This helps explain why easy tasks need extra checking.
- The classifier uses information available before code generation. It does not need to see whether the program succeeds or fails.
- The rules use the same prompt to address false assumptions. This makes it easier to judge the benefit of choosing different tasks.
- The study tests false statements across several models and types of assumptions. This gives a broader value of the problem.
Critical weaknesses
- All inputs in the routing study contain false sortedness claims. Real requests can contain valid guarantees too; the study does not test when distrusting those guarantees adds needless work or changes the intended solution.
- The predictor learns lexical cues for sortedness, uniqueness, and sign constraints. Task-held-out testing does not show that these cues transfer to new false premises or unfamiliar wording.
- The intervention changes a prompt, and the budget counts intervention calls. As Appendix S3 states, tokens and compute are not matched, so the result supports prompt routing more directly than efficient reasoning scaling.
- The organization and formatting of the Related Work section could benefit from further refinement, as the current presentation makes it somewhat challenging for readers to follow the progression of research and recent developments in this field.
Questions for authors
Please respond to the concerns raised in Critical Weaknesses.
Rebuttal for Reviewer aV1j
We thank Reviewer aV1j for the constructive feedback and for recognizing specification-conditioned vulnerability as distinct from task difficulty. We particularly appreciate the reviewer noting that pre-generation features enable efficient routing, and that a shared mitigation prompt isolates the value of instance selection from mitigation effectiveness. We address the reviewer’s concerns regarding empirical robustness, evaluation rigor, and presentation in detail below.
Comment 1: Valid premises and unnecessary intervention
Authors’ response:
We evaluated our intervention on tasks with correct premises, yielding two key observations:
Table R3: Pass@1 under valid premises.
Comment 2: Transfer across premise wording
Authors’ response:
To prove our predictor does not overfit to specific lexical cues, we rigorously tested its robustness to unfamiliar wording:
Table R4: Change in Capture@20% under alternative premise wordings, relative to the original (pp).
Comment 3: Intervention-call budget versus compute
Authors’ response:
We provide additional token-level evaluations to address this concern. Under the original fixed 20% intervention-call budget, PSR consumes the fewest tokens (140.5K) while achieving the highest Pass@1 gain (+5.77 pp). More importantly, under an identical aggregate token budget of 149,831 tokens, PSR achieves a +6.08 pp gain, substantially outperforming all alternative allocation policies.
Table R5. Token consumption and Pass@1 gains under fixed-call and matched-token budgets.
The fixed-call comparison uses a 20% intervention-call budget. The matched-token comparison uses an identical aggregate budget of 149,831 tokens across all policies.
Comment 4: Related Work organization
Authors’ response:
We revised the Related Work section to improve readability and logical progression, reorganizing the discussion to connect misleading-premise risks, adaptive inference methods, and our pre-generation vulnerability-routing contribution.
The updated PDF has been revised to reflect these changes.