- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies spending inference-time compute on new executable tests for a fixed pool of candidate answers. Falsification-Time Compute constructs small databases or program inputs that separate candidate behaviors, then uses a trusted checker or a candidate-blind learned adjudicator to decide which behaviors to reject. The framework distinguishes finding a discriminative witness from interpreting it correctly. On the primary 100-task Spider pool, an engineered SQL search policy improves Test Suite Accuracy from 66% to 73%, with an 83% pool oracle. Controlled studies compare candidate-conditioned, shared, and open witness search. A fully specified local proposer–adjudicator pipeline obtains smaller gains on SQL and loses the search benefits on a small code pool, illustrating the limitations of learned semantic authority.
Critical strengths
-
The primary selection improvement survives a meaningful evaluation check. The authors relabel all 562 candidates using the official multi-database Spider test-suite evaluation while keeping the pools and selected candidates fixed. The aggregate result remains seven recoveries and zero regressions, giving a paired improvement of seven percentage points with a 95% interval of two to twelve points. This is considerably stronger evidence than agreement with a reference query on one database. The separation between missing correct candidates and residual selection errors also makes the measured 41.2% oracle-gap closure interpretable.
-
The distinction between search and semantic authority is made operational. The open-search experiments freeze witnesses before reference interpretation, while the learned adjudicator receives anonymous output classes and is blinded to candidate programs and their identities. This helps isolate whether a useful distinction was found and whether the interpreter used it correctly. The code results are especially informative: reference interpretation recovers both available errors, whereas the learned pipeline loses both recoveries and introduces a regression. The paper also correctly limits what union-of-three adjudication guarantees; repeatedly making the same semantic mistake defeats its conservative aggregation. These failure analyses are useful evidence about a genuine obstacle to automated testing with learned oracles.
Critical weaknesses
-
The paper does not establish a substantial methodological advance over active test selection. Prior work already uses generated tests to distinguish a live pool of candidate programs and obtains external semantic feedback before pruning. TiCoder [1], for example, ranks tests by how well they split candidate code suggestions and uses user feedback to establish intended behavior. FTC's SQL worlds and blinded learned authority are useful differences, but the acquire–execute–interpret–prune loop itself has a close precedent. The paper discusses counterexample-guided synthesis and generated testing, yet omits this particularly relevant comparison.
This makes the choice of empirical comparison important. On the primary public-overfit pool under single-database scoring, actively selected shared tests already match family-routed FTC at 83%. The paper appropriately includes shared search within its framework and does not claim that candidate conditioning always wins. However, these comparisons have not been brought under the TSA contract used for the main result. Additional TSA results also show no gain for several Qwen and mixed pools. A common evaluation of shared search, execution consensus, and the proposed policies, with the same authority and comparable budgets, would clarify what the study establishes beyond the value of obtaining additional tests.
-
The strongest results leave central questions about reproducibility, automated performance, and cost unresolved. In the fully specified local SQL pipeline, Public scores 52/80 and learned authority scores 54/80: three recoveries and one regression. Only 16/80 tasks receive a usable nontrivial witness. On the 29-task code pool, learned authority reduces accuracy from 23/29 to 22/29. These small experiments are useful diagnostics, but provide limited evidence for a generally effective automatic inference strategy. Reference-assisted Stage A measures search power, and the paper correctly labels it as such; its gains cannot resolve the learned-authority bottleneck.
Reproducing the larger engineered results is also harder than the detailed artifact audit might suggest. Table 34 states that the full historical solver, judge, falsifier, and repairer model/backend identities are not established. The stronger session-agent proposer and semantic anchor likewise have unresolved identities and are explicitly provisional. Frozen outputs support auditing reported selections, but leave the corresponding inference procedures incompletely specified. Finally, resource accounting consists of partial logs and coarse call counts. On the controlled end-to-end bank, FTC uses more model and execution calls than judge reranking, while reaching higher accuracy. Establishing a practical inference advantage requires a reproducible comparison at a common token or latency budget.
I recommend clear rejection. The primary SQL result is a credible selection improvement, and the semantic-oracle failure analysis is useful. However, the paper does not establish a sufficiently distinct methodological contribution or an inference advantage over relevant alternatives at a comparable budget. The missing TSA comparisons, unresolved provenance of stronger configurations, and limited gains from the fully specified automatic pipeline affect the central case for the method. Addressing these concerns requires substantial additional evidence: the main gains need to be demonstrated with fully specified models and compared with relevant test-selection baselines under a common evaluation and compute budget.
References
[1] S. K. Lahiri et al. Interactive Code Generation via Test-Driven User-Intent Formalization. arXiv:2208.05950, 2022.
Questions for authors
-
Could the frozen shared-search and execution-consensus selections in Table 19 be evaluated under TSA? In particular, do the primary seven recoveries represent improvements over these selectors, and how do their search and authority budgets compare? This would help identify which concrete evidence-acquisition policy contributes the gain.
-
For the primary 100-task family-routed result, could you specify exactly how a witness authorizes rejection, including the applicable executable properties, any learned judgments, and all model/backend identities? An end-to-end trace for the seven recoveries would help distinguish domain-specific checks from learned or reference-assisted authority and establish what can be regenerated.
-
Could you provide a small budget sweep for the named local pipeline, reporting usable-witness coverage, recoveries, regressions, and total tokens or latency? With 26/80 SQL proposals hitting the 1,400-token cap, it would be useful to know how additional search compute affects coverage and whether the adjudicator preserves any resulting gains.