acceptodds
Under review as a conference paper at ICLR 2027

TRUSTCONS: EXACTLY VERIFYING CONSISTENCY, COMPLETENESS , AND CONDITIONED RE-ENUMERATION IN LANGUAGE MODELS

Abstract

Reliable constraint reasoning requires more than a correct satisfiability verdict: a model should recover the target set, preserve answers under meaning-preserving changes, and — when told that a candidate set contains exactly one deterministic error — return the exact target set. We introduce TRUSTCONS, an exhaustive- oracle diagnostic with 48 boundary-selected Boolean instances, five views, and three tasks, plus a matched extension that holds visible formula dimensions fixed while raising the exact solution count from 1 to 32. An exploratory run exposed a completion-budget confound; we froze it, reran complete matrices under larger limits, treated cap-hit unparseable outputs as missing, and reverified every solution set with Z3 5.0.0. Because v2 reissued complete matrices rather than selected cells, 7,344 cells carry both a v1 and a v2 record under one configura- tion, instance, view, task, and prompt, so the correction is measured rather than asserted: the larger caps moved 3,184 censored cells into the observed popula- tion, 2,839 of them to strict correctness, a paired change of 38.4pp (95% base- cluster interval [34.5, 42.1]pp). Unconditional worst/best-case macro ranges are 45.7–68.2%, 50.2–57.8%, and 25.7–64.2% for decision, low-cardinality exact-set recovery, and the candidate-conditioned third task; the pooled point estimates in- side them are record-level averages that include the below-floor cells, and equal- weighting configurations yields different values. Among 192 fully observed de- cision/enumeration pairs from 4 configurations, 5.2% (95% base-cluster interval 2.1–8.9%) are decision-correct but enumeration-wrong. The third task returns the model’s own enumeration set on 89.8% of strictly parsed pairs and, on those 147 parsed pairs, is correct about equally often (7 against 7, exact McNemar p = 1.00; unparseable answers are missing, not incorrect), so it is scored as conditioned re- enumeration, not as evidence of detection, localisation, or repair. TrustCons is a small synthetic diagnostic, not a safety certificate; its contribution is protocol- aware joint measurement and a descriptive stress test of exact-set scaling under matched visible dimensions.

Primary area:
datasets and benchmarks
Keywords:
LLM reasoning evaluation, Boolean satisfiability (SAT) benchmarks, exact solution-set enumeration, invariance under meaning-preserving transformations, benchmark validity and evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_evLnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

TrustCons evaluates logical reasoning through satisfiability decisions, exact solution enumeration, and conditioned re-enumeration under semantic transformations. It also analyzes the impact of token-budget censoring and output-format requirements on evaluation results.

Critical strengths
  1. Exact solution sets and solver verification provide reliable ground truth.
  2. The analysis of token-budget censoring and format sensitivity is systematic and practically useful.
Critical weaknesses
  1. Limited novelty and significance. Existing benchmarks already employ exact verification, solution enumeration, and semantic transformations. The proposed joint evaluation is largely an integration of existing components, while the experiments do not establish sufficiently substantial new insights into reasoning reliability.
  2. Insufficient empirical support. In the primary corpus, 43/48 instances have at most one solution, and the highlighted cross-view ranking reversal relies on only 11 common instances. Moreover, conditioned re-enumeration does not demonstrate a distinct repair capability.
Questions for authors
  1. Beyond formatting and token-budget effects, what novel and robust semantic failure mode does TrustCons reveal that existing benchmarks cannot identify?
  2. Can the authors provide stronger evidence that the joint diagnostic offers meaningful additional value, particularly after excluding interface failures?
Reviewer_SEaNOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TrustCons, a joint evaluation framework for Boolean constraint reasoning that examines satisfiability decisions, complete solution-set recovery, candidate-conditioned re-enumeration, and consistency across renderings. The authors verify target solution sets through exhaustive enumeration and Z3, evaluate multiple model configurations, and analyze how interface requirements, generation budgets, and missing responses affect measured performance. An extension varies the number of solutions.

Critical strengths

Exact scoring against exhaustively enumerable solution sets provides a reliable basis for checking model outputs and distinguishing error types.

Critical weaknesses

The paper has substantial weaknesses in its research design and organization of contributions. It combines several established evaluation objectives within a unified framework, but provides weak justification for their integration and added diagnostic value. The theoretical formalization and methodological design appear pieced together. The experiments focus on relatively simple solution-set structures and provide limited discriminating evidence for the completeness capability emphasized by the paper, leaving the central contribution weakly supported. The main text also devotes extensive space to problems in the initial evaluation setup and subsequent corrections, placing an audit of experimental execution at the center of the study while yielding limited generalizable insight. The volume of material and complexity of presentation are seriously disproportionate to the actual scientific contribution, revealing substantial weaknesses in formulating the research question and organizing the evidence. For a submission aimed at evaluating reliable reasoning, this quality of research design and evidential support is disappointing.

Questions for authors

None. My main concerns are detailed in the weaknesses section.

Reviewer_9XR9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies whether language models can find every solution to a logical problem and remain correct when the same problem is expressed differently. TrustCons tests 48 Boolean formulas in five forms. The tasks ask models to decide whether a solution exists, list all solutions, and recover the full solution set after receiving a candidate set known to contain one error. Gold solutions are verified by exhaustive search and Z3. An extension tests solution sets of 1 to 32 items. The study also examines how token limits and unfinished responses affect the results.

Critical strengths

Exhaustive search and an independent solver check provide a reliable basis for scoring answers. The distinction between answer agreement and correctness is also useful: a model can give consistent but wrong answers. The authors openly report problems in the initial protocol, rerun complete evaluation matrices, and account for missing responses. This makes the results easier to assess.

Critical weaknesses
  1. The rerun improves the initial protocol, but many responses still reach the token limit without producing a parseable answer. Across v2, this affects 2,549 of 7,344 calls (Table 24). The authors appropriately treat these responses as missing and report accuracy ranges that allow for either correct or incorrect missing answers. However, the unconditional strict-accuracy bounds remain wide: 45.7-68.2% for zero-shot base decision and 25.7-64.2% for the candidate task (Section 6.1; Table 2). These ranges describe the uncertainty; they do not establish which comparisons would hold with enough budget for responses to finish. The problem is especially clear in the tests of equivalent formulations. For the two DeepSeek settings with high reasoning effort, only 12 and 15 of 48 formulas have observed decision responses in all five forms. The bounds for correctness across all five forms are 31.2-100% for V4 Flash and 25.0-100% for V4 Pro (Table 3), leaving their robustness ordering unresolved. The illustrative ranking reversal rests on just 11 shared formulas (Table 3; Section 6.2). The no-thinking settings have better coverage and provide valid evidence that agreement can coexist with errors. The high-reasoning comparisons need validation across larger token budgets; full coverage of all five forms also remains limited to DeepSeek.

  2. A central motivation is to detect missing solutions that a SAT/UNSAT verdict would overlook. Yet 43 of the 48 main formulas have at most one solution, and none has four or more (Section 3). The extension addresses this limitation and includes repeated runs: Gemini shows a roughly 30.4-percentage-point paired decline from one to 32 solutions (Table 12). This is a useful result. The extension, however, covers only four configurations on one eight-variable 2-CNF construction. Evidence about recovery of larger solution sets therefore remains narrow.

  3. Providing an incorrect candidate set could reveal behavior that ordinary enumeration misses. The current results do not establish that added value. Of 147 pairs in which both answers pass strict parsing, 132 return the same set; each task is uniquely correct on seven pairs (Table 25). This does not prove the tasks equivalent, but shows no net accuracy gain from candidate conditioning on this subset. A matched ablation varying candidate content would clarify whether the extra task reveals useful behavior beyond re-solving the formula.

The incremental contribution needs stronger justification against the closest prior work. LogiConBench (ICLR 2026) already evaluates exhaustive Boolean-label enumeration and separates format accuracy from exact-set accuracy. MuSoBench (Guan et al., AAAI 2026) studies complete solution recovery, solution-count strata, and the correction and discovery of solutions. Ghosh et al. (ICLR 2025) separately evaluate logical consistency and accuracy under equivalent expressions. TrustCons brings related measurements into a common framework with censoring analysis, but the current experiments do not establish a sufficiently strong new finding from this integration.

Minor presentation issue. Some labels in Figure 1 overlap. This does not prevent understanding the main content, but it reduces visual clarity and detracts from the overall presentation. Please adjust the label spacing and font sizes.

Questions for authors
  1. Do the main comparisons remain stable as token budgets increase and more responses finish? Where supported, a structured-output control would also help assess formatting effects.
  2. Mistral scores 0/48 on enumeration even with lenient extraction. Why does it also fail on the 24 UNSAT cases, where the correct solution set is empty? Please show representative raw responses.
  3. Is there an ablation or response-level analysis showing what the candidate set changes beyond ordinary enumeration?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.