acceptodds
Under review as a conference paper at ICLR 2027

FALSIFICATION-TIME COMPUTE: SCALING INFER-ENCE BY SEARCHING FOR COUNTEREXAMPLES

Abstract

Additional inference-time compute can change the evidence available for choos-ing an answer, rather than generate another answer. Falsiffcation-Time Compute (FTC) actively acquires executable worlds—witnesses—that distinguish surviving candidate answers. It separates two questions: can search ffnd a useful disagree-ment, and what semantic authority justiffes acting on it? On 100 Spider tasks with frozen model-generated candidate pools constructed to pass weak public ex-ecution tests, FTC improves selection from 66% to 73% under true Test Suite Accuracy (TSA), against an 83% pool oracle, closing 41.2% of the measured gap. Controlled and natural-pool comparisons show that candidate conditioning strengthens weak proposals, while sufffciently strong shared search can also ffnd useful distinctions. Open witness search retains positive selection value under TSA without explicit hand-written failure-family routing. Yet ffnding evidence does not ensure its correct use: candidate-blind learned adjudication introduces an additional regression in a fully local SQL conffguration and loses both avail-able search recoveries in a small code diagnostic. FTC treats executable evidence discovery and the semantic authority needed to use it as distinct inference-time computational problems.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Falsification-Time Compute, Executable Witnesses, Semantic Authority

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mJjJOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies spending inference-time compute on new executable tests for a fixed pool of candidate answers. Falsification-Time Compute constructs small databases or program inputs that separate candidate behaviors, then uses a trusted checker or a candidate-blind learned adjudicator to decide which behaviors to reject. The framework distinguishes finding a discriminative witness from interpreting it correctly. On the primary 100-task Spider pool, an engineered SQL search policy improves Test Suite Accuracy from 66% to 73%, with an 83% pool oracle. Controlled studies compare candidate-conditioned, shared, and open witness search. A fully specified local proposer–adjudicator pipeline obtains smaller gains on SQL and loses the search benefits on a small code pool, illustrating the limitations of learned semantic authority.

Critical strengths
  1. The primary selection improvement survives a meaningful evaluation check. The authors relabel all 562 candidates using the official multi-database Spider test-suite evaluation while keeping the pools and selected candidates fixed. The aggregate result remains seven recoveries and zero regressions, giving a paired improvement of seven percentage points with a 95% interval of two to twelve points. This is considerably stronger evidence than agreement with a reference query on one database. The separation between missing correct candidates and residual selection errors also makes the measured 41.2% oracle-gap closure interpretable.

  2. The distinction between search and semantic authority is made operational. The open-search experiments freeze witnesses before reference interpretation, while the learned adjudicator receives anonymous output classes and is blinded to candidate programs and their identities. This helps isolate whether a useful distinction was found and whether the interpreter used it correctly. The code results are especially informative: reference interpretation recovers both available errors, whereas the learned pipeline loses both recoveries and introduces a regression. The paper also correctly limits what union-of-three adjudication guarantees; repeatedly making the same semantic mistake defeats its conservative aggregation. These failure analyses are useful evidence about a genuine obstacle to automated testing with learned oracles.

Critical weaknesses
  1. The paper does not establish a substantial methodological advance over active test selection. Prior work already uses generated tests to distinguish a live pool of candidate programs and obtains external semantic feedback before pruning. TiCoder [1], for example, ranks tests by how well they split candidate code suggestions and uses user feedback to establish intended behavior. FTC's SQL worlds and blinded learned authority are useful differences, but the acquire–execute–interpret–prune loop itself has a close precedent. The paper discusses counterexample-guided synthesis and generated testing, yet omits this particularly relevant comparison.

    This makes the choice of empirical comparison important. On the primary public-overfit pool under single-database scoring, actively selected shared tests already match family-routed FTC at 83%. The paper appropriately includes shared search within its framework and does not claim that candidate conditioning always wins. However, these comparisons have not been brought under the TSA contract used for the main result. Additional TSA results also show no gain for several Qwen and mixed pools. A common evaluation of shared search, execution consensus, and the proposed policies, with the same authority and comparable budgets, would clarify what the study establishes beyond the value of obtaining additional tests.

  2. The strongest results leave central questions about reproducibility, automated performance, and cost unresolved. In the fully specified local SQL pipeline, Public scores 52/80 and learned authority scores 54/80: three recoveries and one regression. Only 16/80 tasks receive a usable nontrivial witness. On the 29-task code pool, learned authority reduces accuracy from 23/29 to 22/29. These small experiments are useful diagnostics, but provide limited evidence for a generally effective automatic inference strategy. Reference-assisted Stage A measures search power, and the paper correctly labels it as such; its gains cannot resolve the learned-authority bottleneck.

    Reproducing the larger engineered results is also harder than the detailed artifact audit might suggest. Table 34 states that the full historical solver, judge, falsifier, and repairer model/backend identities are not established. The stronger session-agent proposer and semantic anchor likewise have unresolved identities and are explicitly provisional. Frozen outputs support auditing reported selections, but leave the corresponding inference procedures incompletely specified. Finally, resource accounting consists of partial logs and coarse call counts. On the controlled end-to-end bank, FTC uses more model and execution calls than judge reranking, while reaching higher accuracy. Establishing a practical inference advantage requires a reproducible comparison at a common token or latency budget.

I recommend clear rejection. The primary SQL result is a credible selection improvement, and the semantic-oracle failure analysis is useful. However, the paper does not establish a sufficiently distinct methodological contribution or an inference advantage over relevant alternatives at a comparable budget. The missing TSA comparisons, unresolved provenance of stronger configurations, and limited gains from the fully specified automatic pipeline affect the central case for the method. Addressing these concerns requires substantial additional evidence: the main gains need to be demonstrated with fully specified models and compared with relevant test-selection baselines under a common evaluation and compute budget.

References

[1] S. K. Lahiri et al. Interactive Code Generation via Test-Driven User-Intent Formalization. arXiv:2208.05950, 2022.

Questions for authors
  1. Could the frozen shared-search and execution-consensus selections in Table 19 be evaluated under TSA? In particular, do the primary seven recoveries represent improvements over these selectors, and how do their search and authority budgets compare? This would help identify which concrete evidence-acquisition policy contributes the gain.

  2. For the primary 100-task family-routed result, could you specify exactly how a witness authorizes rejection, including the applicable executable properties, any learned judgments, and all model/backend identities? An end-to-end trace for the seven recoveries would help distinguish domain-specific checks from learned or reference-assisted authority and establish what can be regenerated.

  3. Could you provide a small budget sweep for the named local pipeline, reporting usable-witness coverage, recoveries, regressions, and total tokens or latency? With 26/80 SQL proposals hitting the 1,400-token cap, it would be useful to know how additional search compute affects coverage and whether the adjudicator preserves any resulting gains.

Reviewer_Zc2EOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces an inference-time framework that constructs inputs or databases to distinguish candidate programs or SQL queries. Candidates are grouped by their execution outputs, and a trusted checker or learned adjudicator determines which groups to retain. Experiments examine engineered and model-generated tests, along with reference-assisted and learned interpretation on SQL and code tasks.

Critical strengths
  1. The paper distinguishes execution disagreement from evidence that a candidate is wrong.
  2. Freezing candidate pools helps isolate the effect of additional testing. The main SQL result also uses multi-database evaluation rather than agreement on a single database.
  3. The authors report negative results and acknowledge limitations, including failures of learned adjudication and incomplete compute comparisons.
Critical weaknesses
  1. The novelty of the paper is very limited. Constructing inputs that distinguish candidate programs and consulting an oracle for the intended output are established ideas in oracle-guided synthesis and testing. The proposed workflow mainly combines test generation, execution-based grouping, a simple split score, and model-based adjudication. The paper does not clearly establish what technical advance this combination provides over existing distinguishing-input methods or generated-test selection approaches.

  2. Although the paper introduce the reliability problem, it is unclear how it can be resolved by the proposed method. The framework can expose differences between candidates, but determining which behavior satisfies the task still depends on a trusted reference, hand-designed properties, or another model. The reference-assisted experiments therefore demonstrate useful test discovery, not a reliable reference-free selection method. With learned adjudication, the SQL improvement is small, and code accuracy falls below the baseline. The distinction between search and interpretation is reasonable, but the paper offers limited progress on the difficult part.

  3. The experiments do not establish the value of the proposed use of computation. There is no controlled comparison against spending comparable resources on additional candidates, stronger judging, or established test-generation methods. The compute-allocation sweep changes search policies alongside allocation, making its results difficult to interpret as evidence of scaling. Shared tests also match or exceed family-routed search in several reported settings, leaving the advantage of the proposed candidate-directed approach unclear.

  4. The experimental evidence is fragmented and narrow. The paper moves between controlled tasks, natural candidate pools, engineered search, open search, and different interpretation mechanisms without establishing one consistently evaluated method. Several additional pools show little or no improvement. The controlled bank contains 176 variants of only 44 base tasks, while the code study has just two recoverable baseline errors. These results do not convincingly establish a general benefit.

  5. Important parts of the evaluation remain difficult to reproduce and interpret. The paper leaves the model identities of some stronger components and historical configurations unresolved. Comparisons also use different scoring protocols, and several stronger baselines are not included in the main TSA evaluation. And the extensive historical results and protocol corrections make it difficult to trace which implementation supports each claim.

Questions for authors
  1. Could the authors give a concrete example where the proposed search policy provides an advantage over an equally resourced task-conditioned test generator, and explain which mechanism produces that advantage?
  2. What exact procedure authorizes candidate rejection in the headline 66% to 73% experiment? Please clarify which task-specific properties or reference information it uses and how those properties were validated.
  3. Which main conclusions remain supported when considering only configurations with fully specified model identities and the same evaluation protocol?
Reviewer_o9ysOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies and proposes falsification time compute. Here, instead of generating more solutions, the system searches for counterexamples for distinguishing candidate solutions. Authors study this in SQL and code tasks and show that generating such executable tests can improve candidate selection.

Critical strengths

Using additional inference-time compute for falsification instead of generation is an interesting direction. The distinction between finding counterexamples and having enough semantic information to reject a candidate makes sense.

Critical weaknesses
  1. Novelty of this approach is unclear. Given extensive work on test generation and counterexample-guided selection. For example, S* (https://arxiv.org/abs/2502.14382) already proposes generating distinguishing test inputs for pairs of candidate programs and using execution feedback for selection. Similarly, CodeT and CEGIS-based methods have studied related ideas. While the framing around falsification-time compute is interesting, what new learning does FTC bring beyond these existing approaches?
  2. The empirical evidence seems somewhat limited. The main evaluation uses only 100 Spider tasks, and several other natural candidate pools show little or no improvement. Controlled experiments use augmented variants from only 44 base tasks. It is unclear how well FTC generalizes to other models and domains. Can the authors provide more extensive evaluations across different candidate pools and tasks?
  3. The open-search experiments use reference-based interpretation, which is not available in realistic inference settings. With learned interpretation, performance drops, sometimes even below the original baseline. While the engineered approach performs better, it is unclear how easily this generalizes to new domains without hand-designed semantic checks. Additionally, can the authors compare against other verification-based selection approaches such as V1 (https://arxiv.org/abs/2603.04304) and LLM-as-a-Verifier (https://llm-as-a-verifier.com/), under matched candidate pools and inference-time compute? Prior work such as When To Solve, When To Verify (https://arxiv.org/abs/2504.01005) has also shown that spending more compute on generating solutions can be more efficient than verification. It would be useful to study whether spending additional compute on falsification is actually better than generating more candidates or using stronger verification.
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.