acceptodds
Withdrawn from ICLR 2027

Symbolic Reasoning Groundings for Latent Visual Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Neuro-symbolic learning combines flexible neural perception with structured, executable reasoning. Yet systems that place a predefined executor on the inference path bind their answers to a fixed ontology, limiting integration with open-ended vision–language models. We introduce neuro-symbolic supervision for latent reasoning: a framework that trains over continuous reasoning states by grounding them as shared entities and executable predicates. Symbolic losses thereby shape the latent trajectory from which the original VLM backbone generates its answer. This brings compositional supervision into latent reasoning while allowing the decoder to answer beyond the symbolic vocabulary. We instantiate the framework in through local symbolic grounding, compositional symbolic grounding, and reinforcement through discrete symbolic execution. The final stage pairs each sampled text with one symbolic scene and optimizes both using answer and execution rewards. The symbolic path is used only during training. With four latent steps, reaches a 72.53% mean across six benchmarks, achieving state-of-the-art performance.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
symbolic reasoning, latent reasoning, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nLV2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Sylva, a neuro-symbolic supervision system for latent reasoning vision language models. This is a 3 staged training approach which compositionally builds symbolic meaning into latent tokens from local grounding to latent chain consistency to full execution accuracy. Their first stage maps knowledge triplets into each latent step. The second stage then clusters triplets into typed probabilistic scene graph which executes the supervising first order logic program differentiably. The last stage performs a GRPO-style reinforcement learning on the final answers by sampling the scene graph and text and rewarding both with a weighted reward. They report state of the approach results on 5 our of 6 benchmarks evaluated.

Critical strengths

The three stages all have reasonable justifications and designs. It logically follows that each stage ought to follow the previous, and that the differentiable FOL execution and the discretely sampled RL stages would compliment each other. The explanations are lucid and the figures provide appropriate context.

While many of the components of this submission are closely related to previous work, the combination of techniques are novel. The use of Hungarian matching for stage 2 label generation in latent VLMs is a meaningful contribution. I can appreciate the use of sampling in stage 3 to preserve boolean semantics necessary for RL, because it may be tempting to attempt to maintain the soft logic approach in stage 2 into stage 3.

Critical weaknesses

My main concern is the lack of out-of-distribution evaluation. The problem with approaches like Sylva tend to be overfitting to the domain and heavily leaning on the ground truth data's ontologies. It would be beneficial to the defense of the claims to demonstrate that these results hold on benchmarks or entities/predicates/operators outside of the training data.

Stage 1 is essentially the same as the cited ReGuLaR approach. The question then becomes "is the performance advantage from the novel training, or simply additional training?" I recommend evaluating an additional baseline that continues ordinary SFT from stage 1 without either symbolic loss. That would more cleanly isolate the performance effect between the training budget and the training objective.

Two similar approaches deserve a citation in your related works: SAGE https://arxiv.org/abs/2607.05716 and APEIRA https://arxiv.org/abs/2606.01215

It would be nice to see some sort of intervention or experiment to demonstrate that the VLM's language decoder is actually using the symbolically enhanced latent representations to carry out logical operations. Showing a controllable/steerable manipulation effect on one of these latent steps would provide good evidence that the claimed symbolic reasoning is actually happening, rather than focusing solely on the improved performance.

Questions for authors

Given the reliance on annotated entities, relations, and FOL programs, can the authors provide an out-of-distribution evaluation, particularly on held-out combinations of concepts and relations or benchmarks outside the training distributions? This would help establish whether the gains reflect generalizable compositional reasoning rather than adaptation to the training ontology.

Can the authors evaluate a baseline that continues ordinary SFT from the Stage 1 checkpoint, using the same Stage 2 data and training budget, but without either symbolic loss? This would help isolate the contribution of the proposed symbolic objectives from the benefits of additional fine-tuning.

Since the symbolic executor is removed during inference, is there evidence that the language decoder actually uses the symbolically grounded latent representations for logical reasoning? Could the authors perform a targeted intervention, such as manipulating an entity or relation representation in a latent step, and demonstrate a corresponding, predictable change in the generated answer? This would help distinguish learned symbolic reasoning from improvements due to auxiliary representation supervision.

Reviewer_C3G8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SYLVA, a framework that uses neuro-symbolic supervision to improve latent visual reasoning. It trains a VLM through three stages: local grounding, compositional symbolic grounding, and reinforcement learning with discrete symbolic execution rewards. The goal is to improve reasoning at inference without explicit symbolic execution. Experiments on six visual benchmarks report improvements over selected baselines, but comparisons with stronger recent models and evidence for compositional reasoning remain limited. Related ideas in program distillation and neuro-symbolic reasoning appeared in earlier works, including work from two years ago, but the paper does not sufficiently discuss these works or clearly explain its new contribution.

Critical strengths

The paper provides stage-wise ablations and component comparisons to study the training framework. The results suggest that symbolic supervision can improve the selected backbone without requiring a symbolic executor at inference. However, similar findings have been reported in previous works.

Critical weaknesses
  1. The technical contribution and reasoning claims are not sufficiently established. The core ideas and concepts do not clearly distinguish its contribution from exisiting visual program distillation and neuro-symbolic reasoning methods. The experiments show benchmark improvements, but do not establish that the decoder uses the learned symbolic structure or generalises to unseen compositions.

  2. Limited comparisons weaken the SOTA claim. The paper mainly compares against older general-purpose models, while many stronger recent VLMs, tool-use agents, agentic VLMs, and relevant distillation-based VLMs are missing. The current results do not establish a clear advantage over these alternatives.

Questions for authors
  1. Why does Table 1 use older general-purpose models while omitting stronger recent models and agentic VLMs released in 2025 and 2026? Models such as Qwen3-VL-8B and Imagine-OPD-4B report comparable or even better results on V*, HRBench-4K, and HRBench-8K. Although evaluation settings need to be aligned, these results raise concerns about the broad SOTA claim.

  2. Visual Program Distillation (CVPR24) already transfers tools and programmatic reasoning into VLMs without requiring those tools at inference. NAVER (ICCV25) also studies compositional visual grounding with explicit probabilistic logic. DWIM (ICCV25) studies tool-use agent with instruction-tuning to enhance compositional visual grounding and reasoning. How does SYLVA differ from these works, and why are they not discussed? More comparison and experiment results are needed.

  3. How does SYLVA compare with multi-tool visual reasoning agents, such as Metis?

  4. What evidence shows that the decoder actually uses the learned entity bindings and symbolic relations? I can see Figure 5 is showing that information recovered by symbolic branch, but this might not establish that the decoder itself relies on it. Can the authors provide intervention experiments and tests on unseen relation combinations or longer reasoning chains?

  5. In appendix B.2, there is a statement that: Possible image overlap with the evaluation benchmarks has not yet been fully audited. Could you please complete this audit? Without this check, the generalisation claims remain uncertain.

Reviewer_nvSrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SYLVA, a neuro-symbolic framework for latent visual reasoning. Symbolic grounding and execution are used only during training, while inference relies on the original VLM decoder. The method combines local grounding, compositional FOL-based supervision, and execution-guided RL, achieving strong results across six benchmarks.

Critical strengths
  1. Interesting idea of using symbolic reasoning as training supervision rather than an inference-time executor.
  2. The method is technically well developed, combining grounding, entity composition, FOL execution, and RL.
  3. Strong empirical performance, outperforming prior latent-reasoning baselines on the reported benchmark average.
Critical weaknesses
  1. Potential train–test image overlap is not fully audited. The paper explicitly acknowledges this issue, which may affect the reliability of the reported benchmark gains.

  2. The contribution of symbolic reasoning is not fully isolated from richer supervision. SYLVA uses bounding boxes, entity annotations, relation labels, entity counts, and FOL programs. A stronger matched baseline with the same supervision but without symbolic execution would better isolate the benefit of the proposed method.

  3. There is limited direct evidence that the latent states themselves become symbolically structured. Most evidence comes from downstream accuracy, while intrinsic evaluations of grounding, entity linking, or symbolic execution accuracy are missing.

  4. The generated FOL programs are validated mainly by execution consistency, not semantic faithfulness. Since program generation uses both the canonical answer and scene annotations, a manual audit would help verify that the programs truly reflect the original questions.

Questions for authors

Please address the weakness 1-4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.