acceptodds
Withdrawn from ICLR 2027

Shuqi: Support-Aware Executable Reasoning for Representation–Execution Diagnosis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Final-answer accuracy reveals whether a reasoning system is wrong, but not whether the error arises from constructing a usable representation or from executing semantics that are already explicit. We study this distinction with a controlled Raw-NL → Same-IR → deterministic diagnostic interface and semantic-stability audits that separately expose representation-associated recovery, residual execution-side error, and instability under fixed semantic content. These diagnostics reveal substantial effects: residual execution gaps reach 23.0 percentage points, while lossless serializations of identical canonical representations disagree on up to 30.8% of disjoint-source examples. Based on these findings, we propose Shuqi, a task-grounded support-aware executable reasoning system that compiles supported inputs into executable representations, performs deterministic task-defined execution with replay-checked evidence, and releases the symbolic result only when a task-specific support contract succeeds; otherwise, it falls back to a frozen generative reasoner. On ProofWriter, Shuqi preserves all 41 symbolic fixes while introducing no regressions, reaching 98.0% accuracy, and achieving 100% on both ProntoQA and LogicalDeduction. These results show that reliable reasoning requires controlling not only what semantic representation is constructed, but also how that representation is executed and when an executable result should be released.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
large language models, logical reasoning, neuro-symbolic reasoning, executable reasoning, representation–execution diagnosis, semantic stability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PQU1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SHUQI, a neuro-symbolic reasoning framework that separates natural-language representation construction from execution. The authors propose a controlled Raw-NL → Same-IR → deterministic execution methodology to study whether reasoning failures originate from representation construction or from executing already explicit semantics.

SHUQI uses task-specific deterministic compilers, symbolic execution, replay-checked certificates, and support-aware fallback to a generative model. The evaluation covers ProofWriter, ProntoQA, and LogicalDeduction, achieving 98%, 100%, and 100% accuracy, respectively. The authors also investigate semantic stability under representation-preserving transformations and different serializations of identical intermediate representations.

The central contribution is a framework for diagnosing representation versus execution failures and selectively using deterministic reasoning when the symbolic representation is supported.

Critical strengths

1. A useful distinction between representation and execution errors.

I really like the motivation behind this paper :) We often evaluate reasoning systems through final-answer accuracy, but this tells us little about whether the model misunderstood the problem or failed to reason correctly about an otherwise accurate representation. The Raw-NL → Same-IR → deterministic comparison provides an interesting way to study this distinction. I particularly appreciate the experiments showing that identical canonical IRs can produce different answers depending on their serialization.

2. Careful treatment of symbolic execution and its limitations.

The support-aware routing mechanism is interesting, especially the ProofWriter experiment where forcing symbolic execution introduces 19 regressions, while SHUQI avoids these by falling back to the original model. I also appreciate the replay-certificate checks and independent bounded execution audits. The authors are relatively careful about distinguishing execution correctness from whether the original natural language was represented faithfully.

Critical weaknesses

1. The strongest results rely heavily on task-specific engineering rather than general reasoning capabilities.

My main concern is that SHUQI relies on deterministic, manually engineered parsers for restricted natural-language templates. On ProntoQA and LogicalDeduction, these parsers cover every evaluated example, meaning that once the input is compiled, the remaining task is handled entirely through deterministic symbolic execution.

This makes the 100% results less surprising than they initially appear. The difficult part of general-purpose reasoning is often constructing the correct formal representation from ambiguous or unfamiliar language. Here, much of that challenge is avoided through predefined grammars and task-specific compilation rules.

I think this is an important limitation because the paper motivates SHUQI as a framework for reliable reasoning more broadly. I would have liked to see experiments with paraphrased inputs, unseen linguistic structures, or learned language-to-IR translation to evaluate whether the approach remains effective when representation construction is not essentially predetermined.

2. The representation–execution comparison does not completely isolate the sources of reasoning error.

I find the diagnostic methodology interesting, but I am not fully convinced that the Raw-NL → Same-IR improvement can be attributed specifically to better semantic representation. The Same-IR condition also changes the input format, removes natural-language complexity, and presents the problem in a structured representation that may be easier for the model to process.

The authors appropriately avoid claiming a causal decomposition, but this still limits how strongly the results can be interpreted. Interestingly, Appendix M shows that after controlling the answer format for Mistral-Nemo, Same-IR does not yield a statistically significant improvement on either task. This suggests that the representation-associated gains may be highly model- and interface-dependent.

I would have liked stronger controls separating semantic content from presentation effects, particularly since identifying these distinct failure modes is one of the paper's main contributions.

3. Limited benchmark diversity and unclear generalization.

The experiments cover only ProofWriter, ProntoQA, and LogicalDeduction, all of which involve relatively constrained symbolic reasoning. These are reasonable diagnostic benchmarks, but I wonder how well the conclusions transfer to mathematical reasoning, program verification, or longer-horizon problems where constructing the representation is itself difficult :)

For example, Lean-based verification or program synthesis would introduce dependencies, auxiliary lemmas, evolving specifications, and incomplete formalizations. The paper does not establish whether its support-aware interface remains useful when a deterministic compiler cannot simply recognize most inputs through fixed templates.

I also think the comparison in Table 2 needs careful interpretation. Although the baselines use the same Qwen3-14B checkpoint, they do not share matched inference resources, and some reproduced methods have substantial invalid-output rates. Consequently, the end-to-end improvements do not isolate the benefit of support-aware routing relative to other well-engineered symbolic systems.

Questions for authors
  1. Generalization beyond templates: How does SHUQI perform on paraphrased or unseen natural-language constructions outside its supported grammars? I am particularly curious how quickly symbolic coverage deteriorates under realistic linguistic variation :)

  2. Representation fidelity: The replay certificates verify execution relative to the compiled IR, but who verifies that the IR actually captures the intended meaning of the original input? Could the authors explore independently checking this translation, perhaps through formal specifications or proof-assistant-based verification?

  3. Benchmark diversity: Have the authors considered extending the evaluation to mathematical reasoning, program verification, or longer-horizon reasoning benchmarks? These seem especially relevant when the representation itself cannot be constructed using fixed templates.

  4. Representation versus formatting: Appendix M shows that the Same-IR improvement disappears for Mistral-Nemo under matched answer-format constraints. How much of the representation-associated improvement for Qwen and DeepSeek remains after applying analogous controls?

  5. Routing generalization: On ProofWriter, all 23 fallback cases arise from incomplete compilation. Would the same routing guarantees hold on unseen inputs where compilation succeeds but produces an incorrect semantic representation?

  6. Comparison with simpler symbolic baselines: Since the supported tasks can be solved deterministically after template parsing, how much does SHUQI improve over a conventional parser-plus-solver system with a simple fallback mechanism? Which parts of the framework are essential beyond this standard architecture?

  7. Scalability: How does the approach scale when the number of entities, rules, or possible symbolic states increases substantially? I would be interested in both compiler coverage and execution overhead on larger instances.

Reviewer_Km1VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how input representation and execution affect reasoning performance. The authors compare generation from natural language, generation from a lossless serialization of canonical intermediate representations, and deterministic execution of the same representations. Additional audits examine responses to meaning-preserving transformations, semantic changes, and different serializations of identical representations. The proposed system, SHUQI, uses task-specific template parsers to construct executable representations, performs deterministic execution, and checks the resulting evidence. It releases the symbolic answer when a task-specific support contract is satisfied and otherwise falls back to a frozen generative reasoner.

The paper evaluates SHUQI on subsets of ProofWriter, ProntoQA, and LogicalDeduction. The experiments report improved accuracy, remaining generative errors after canonical semantics are supplied, and inconsistent predictions across equivalent presentations. On ProofWriter, support-aware routing retains symbolic corrections while avoiding regressions introduced by forced symbolic execution. Additional controls examine routing at matched coverage, rejection of corrupted execution evidence, and agreement with independent reference semantics.

Critical strengths

S1. The controlled representation–execution interface provides a useful way to examine reasoning failures beyond final-answer accuracy. The Raw-NL and Same-IR comparisons keep the generative executor fixed, while the Same-IR and deterministic comparisons use matched semantic content. The identical-IR audits further show that predictions can change even when the underlying representation is unchanged. These controls help distinguish sensitivity to presentation from errors that remain after task semantics are explicit.

S2. The ProofWriter experiments provide concrete evidence for support-aware routing under incomplete symbolic coverage. The comparison with forced symbolic execution separates helpful corrections from newly introduced errors, while matched-coverage controls examine whether the benefit comes from selecting appropriate cases rather than simply using a particular fallback rate. These results support the practical role of checking executable support before replacing the generative answer.

Critical weaknesses

W1. My main concern is that the methodological contribution beyond task-specific parsing and symbolic execution is not sufficiently established. The deployed compilers recognize predefined controlled-language templates, and all ProntoQA and LogicalDeduction examples are handled entirely by the symbolic path. The high accuracy therefore largely demonstrates the effectiveness of these task-engineered pipelines. Although the representation–execution diagnostics provide useful observations, the experiments offer limited evidence about how the system handles linguistic variation outside the supported templates or transfers to tasks requiring new representations and compilation rules.

W2. Some baseline comparison is substantially affected by parsing and other pipeline failures. Appendix B reports numerous invalid outputs in several reproduced baselines, including failures associated with answer-handling procedures that differ from the released implementation. This makes it difficult to determine how much of SHUQI’s advantage comes from reasoning quality rather than differences in pipeline reliability. Checking the results under the released answer-handling procedures and separating implementation failures from reasoning errors would make the comparison more convincing.

Questions for authors

Q1. What parts of SHUQI can be transferred to a new reasoning task without redesign the compiler and execution rules? Could the authors evaluate the frozen system on meaning-preserving linguistic variations outside its predefined templates and report both symbolic coverage and final accuracy?

Q2. How sensitive are the baseline comparisons to parsing and pipeline failures? For example, how would the SymbCoT results change under its released answer-handling procedure, and how much of the remaining performance gap comes from invalid outputs rather than incorrect valid answers?

Q3. Is there a systematic procedure for designing the task-specific pipelines within SHUQI? Beyond the shared workflow, how are the compiler, executor, and release contract determined, and how should a user identify the appropriate pipeline for a new input in practice?

Reviewer_uXynOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies neural symbolic approaches to prompting LLMs on text-to-logic tasks. They focus on intermediate representations (IRs) expressed in formal logic. The paper introduces three settings:

(1) Raw-NL where the answer is generated given the input text (2) Same-IR: Generative execution when the input is a “canonical semantics” intermediate representation of the input text (3) Deterministic Execution: symbolic execution directly on the structured IR

The paper presents an analysis of errors in LLM IR generation and IR execution by LLMs. The proposed solution (SHUQI) is a pipeline comprised of: (1) a deterministic compiler that translates natural language into task-specific executable IR based on templates; (2) a deterministic executor that uses task-specific solvers; (3) a validator (“Release Contract”); and (4) an LLM fallback which basically runs an LLM directly on the task if one of the prior stages fails.

The experimental results show SHUQI achieves near perfect accuracy (98-100%) on subsets of three logic deduction datasets: ProofWriter, ProntoQA and LogicalDeduction. However, the compilers deployed in SHUQI are three distinct exact-template parsers for each dedicated eval rather than a domain general NL-to-IR component. Furthermore, it relies on separate, task-specific rules for each benchmark (signed rule closure for ProofWriter/ProntoQA and SMT solvers for LogicalDeduction). Thus, SHUQI seems to be a specialized multi-step pipeline tailored to three controlled evaluation benchmarks rather than a general-purpose NL-to-Logic approach. In addition, the paper does not discuss whether SHUQI applies to broader real-world applications where deterministic parsers and bounded solvers are unavailable---making the potential contribution limited.

Lastly, the paper is not well-written and the Figures, Abstract, Intro, Related Work (and all other sections to a lesser degree) are all hard to understand and should be improved. In addition, the paper uses dense, non-standard terminology such as "deterministic diagnostic interface", "semantic-stability audits", "support-aware executable reasoning", "canonical semantics", "replay-checked evidence", and "residual execution-side error". It would improve the paper’s presentation to use less vague terminology throughout the abstract and introduction, or to actually explain these terms in the text.

Critical strengths

The paper studies the effectiveness of symbolic approaches in LLMs.

Critical weaknesses
  1. SHUQI seems to be a specialized multi-step pipeline tailored to three controlled evaluation benchmarks rather than a general-purpose NL-to-Logic approach

  2. The paper does not discuss whether SHUQI applies to broader real-world applications where deterministic parsers and bounded solvers are unavailable

  3. The paper uses different underlying LLMs across tables without a unified setup (Table 2 evaluates SHUQI + baselines using Qwen3-14B while the analysis is done using Qwen2.5-14B and DeepSeek-R1-Distill)

  4. Details of the baseline implementation are vague (Appendix B is not very comprehensive)

  5. The paper presentation is very poor in terms of readability and overall structure (Related Work is disjointed, no Discussion section).

Questions for authors
  1. Is there a methodological difference between SHUQI and the evaluated baselines beyond the fact that SHUQI is fully rule-based?

  2. Why did you not develop a unified SHUQI framework for all three benchmarks?

  3. How were the evaluation examples sampled from each of the three benchmarks? Were they randomly sampled? Why not use the standard test sets for each task?

  4. Can SHUQI be easily extended to tasks beyond text-to-logic?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.