acceptodds
Under review as a conference paper at ICLR 2027

Visual Logical Rule Induction by Bridging Inductive Logic Programming and Multimodal Large Language Models

Abstract

We propose ILP-CoT, a method that bridges Inductive Logic Programming (ILP) and Multimodal Large Language Models (MLLMs) for visual logical rule induction, which involves both discovering logical facts and inducing logical rules from a small number of unstructured visual inputs. This task still remains challenging when solely relying on ILP, due to the requirement of specified background knowledge and high computational cost, or MLLMs, due to the appearance of perceptual hallucinations. Based on the methodology that let MLLM be the input perceptor and rule structure proposer, meanwhile let ILP system serve as the rule inducer and symbolic verifier, our approach automatically builds ILP tasks with pruned search spaces, and utilizes ILP system to output rules built upon rectified logical facts and formal inductive reasoning. Its effectiveness is verified through challenging logical induction benchmarks, as well as a potential application of our approach, namely text-to-image customized generation with rule induction. Our code and data will be open-sourced upon formal release.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
neurosymbolic learning, multimodal large language models, multimodal reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_xsPFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • ILP-CoT combines MLLM-based visual grounding and rule proposals with Metagol-based program search. Failure reflection revises the visual facts and proposed structures, while a support-based selector chooses the final executable rule. The core induction pipeline does not require additional model training.
  • The paper evaluates visual classification on V-LoL, CLEVR-Hans, and CURI-90 with four base MLLMs. It also introduces a small customization dataset and demonstrates how inferred rules can guide image generation.
Critical strengths
  • The division of responsibilities is well motivated. The MLLM handles perception and proposes plausible structures, while the symbolic system searches for and executes concrete programs. This yields inspectable intermediate facts and rules.
  • The pipeline improves over prompting baselines across backbones. ILP-CoT outperforms the three prompting baselines in all twelve model and dataset combinations and reaches 81.25%, 82.55%, and 92.19% with Qwen3-VL-32B. The size of this margin should be read with care because the baselines are close to chance on two datasets (Weakness 4).
  • The component analyses are useful. Table 2 examines both reflection and program selection rather than only the full system. Their effects differ across datasets, which suggests that they address distinct sources of error.
Critical weaknesses
  1. Comparisons with closely related methods are missing.

    • Table 1 compares prompting strategies, and Tables 2 and 3 examine internal components and ILP backends. These controls support the authors' design choices but do not place the method in the literature.
    • Vision-Language Programs and Pix2Code are the most relevant candidates. Both have public code, with CLEVR-Hans support in VLP and CURI evaluation in Pix2Code. Published numbers should not be copied without accounting for differences in support size, metric, perception training, and task split.
    • NEUMANN (Shindo et al., 2024) is cited only in passing. A brief distinction would clarify the novelty.
  2. The baselines do not isolate the benefit of ILP search.

    • According to Appendix A, Code CoT produces both the Prolog rule and the final prediction through the MLLM. The comparison therefore changes both how rules are found and how they are applied.
    • Hypothesis Search (Wang et al., 2024) and hypothesis refinement (Qiu et al., 2024) already let an LLM propose programs that an interpreter executes and refines. A baseline of this kind with the same MLLM and the same extracted facts would identify the added value of Metagol. Model calls, candidate counts, and runtime should also be reported.
  3. The generation results do not yet show an advantage from ILP-based induction.

    • Table 4 evaluates rule descriptions, not generated images. The generation examples only compare prompts with and without the learned rule.
    • A fair test would fix the generator and customization and compare Direct Predict rules with ILP-CoT rules. Otherwise the claims should be limited to an illustrative application.
  4. Baseline strength and evaluation scale are unclear.

    • V-LoL and CLEVR-Hans are binary tasks, yet all 24 baseline results on them lie between 40.22% and 58.97%, and 9 are below 50%. GPT-5.2 with Base CoT reaches only 51.18% on V-LoL. This raises the question of whether the baselines are set up fairly.
    • Each of these two datasets has only three tasks (Section 4.1), with no repeated runs or variance. Most ILP-CoT results on V-LoL (81.25, 70.83, 62.50, 58.33, 54.17) are multiples of 1/48, which suggests about 48 queries in total. A 10-point gap would then correspond to about five images.
    • Support and query counts, seeds, and the number of runs should be stated. The selector tolerance for "close" and "similar" accuracy is also unspecified (Appendix B), although the selector drives large gains in Table 2.
  5. The fact-correction analysis does not support the formal-verification interpretation.

    • Fig. 2 gives no protocol for obtaining ground-truth facts. Its "Extra" category also counts visually correct but irrelevant facts as errors.
    • The text claims that errors consistently decrease, yet CURI Missing stays at 1.15%. The corrections also come from MLLM rechecks during reflection, not from ILP verification.
Questions for authors
  1. Comparison with existing methods
    • Could the authors compare with VLP on CLEVR-Hans and, where compatible, Pix2Code on CURI-90 under identical support and query examples? Differences in supervision or compute should be reported.
  2. Attribution of the improvement
    • How does ILP-CoT compare with MLLM-generated rules executed by an external interpreter on the same facts, with comparable candidate and feedback budgets?
    • What are the model-call and runtime costs of each method?
  3. Evaluation reliability
    • How are unparsable baseline answers handled, and are the query sets class-balanced?
    • How many queries does each task contain, and how stable are the results across runs?
    • What tolerances does the selector use, and how were the ground-truth facts in Fig. 2 obtained?
  4. Downstream evidence
    • Does an ILP-CoT rule improve the generated image over a directly inferred rule under the same generation setup?

Limitations

  • Consistency is not a guarantee of the intended rule. A program may fit the extracted facts and support labels through an unintended correlation. Formal verification claims should be limited to consistency with the supplied representation and examples.
  • The generation application is preliminary. It works as a demonstration but does not establish superiority in relation-aware generation.

  • The pipeline is promising, but the evaluation relies on prompting controls that are close to chance and on internal substitutions. The added value over existing approaches is therefore hard to judge.
  • I would raise my score if fair comparisons with closely related methods show a meaningful benefit and the evaluation protocol is made explicit. I do not expect an exhaustive benchmark suite or superiority in every setting. The priority is evidence that the integration helps beyond stronger prompting, external execution, or additional search.
2 comments

We would like to thank the reviewer for the detailed comments and suggestions. We appreciate your recognition that our control experiments support our design choices.

1. Comparison with existing visual rule learners

VLP and Pix2Code. Both used ILP-CoT's support and query images and the same visual extractor, with no additional perception training. With Qwen3-VL-32B, VLP reached 64.41% versus 86.72% on CLEVR-Hans; Pix2Code reached a preliminary 68.00% versus 92.19% on CURI-90 (Table R8). Table R3 covers all four VLP base models.

NEUMANN. It learns clause weights; Metagol searches for predicate assignments. Table R1 compares both approaches and αILP.

Table R8. Requested baselines with Qwen3-VL-32B. Query accuracy in % (paper Table 1).

DatasetVLPPix2CodeHypothesis SearchHypothesis refinementILP-CoT
V-LoL—50.00 ± 0.0051.82 ± 0.9850.13 ± 3.0883.20 ± 3.20
CLEVR-Hans64.41 ± 1.9250.00 ± 0.0063.11 ± 3.2669.53 ± 3.4186.72 ± 0.52
CURI-90—68.0068.72 ± 0.9367.88 ± 2.3192.19 ± 2.68

± denotes the standard deviation across seeds. Dashes indicate ongoing results. Pix2Code induced no rule on V-LoL or CLEVR-Hans. Its CURI-90 score is preliminary, with further runs pending.

2. Attribution of the improvement and computational cost

What is matched? Hypothesis Search and hypothesis refinement execute proposals on the same initial facts and examples, with the same reflection cap and scoring (Appendix A; Table R8). The original Code CoT prompt required SWI-Prolog execution. Revised Code CoT replaces Metagol search with MLLM-written code. Both methods select rules on the same support examples and execute them on the same query facts. On CLEVR-Hans, Code CoT evaluated 10.33 rules per task and reached 68.49% accuracy. ILP-CoT evaluated 3,267 and reached 86.72% (Qwen3-VL-32B, Table R4).

Calls and runtime. Table R9 compares successful MLLM calls and rule-learning time for all four methods in paper Table 5A. Table R4 separately counts calls used to generate candidates.

Table R9. Computational cost with Qwen3-VL-32B, mean per task (paper Table 5A).

MethodV-LoL callsV-LoL minutesCLEVR-Hans callsCLEVR-Hans minutesCURI-90 callsCURI-90 minutes
Code CoT622.671.48195.330.7925.171.03
ILP-CoT623.1118.16195.674.5523.2716.89
Hypothesis Search534.560.73354.330.73200.940.93
Hypothesis refinement291.220.29108.780.0952.780.15

Calls = successful MLLM calls; minutes = rule-learning time after fact extraction. Paper Table 5B counts each shared request and its tokens once across ILP-CoT, Code CoT, NEUMANN and αILP.

3. Evaluation reliability

Query counts and class balance. V-LoL now has 128 positive and 128 negative queries per rule, replacing the original 48-query evaluation. CLEVR-Hans has 64 of each per concept (384 pairs; 192 distinct images). CURI-90 has 301 positive and 1,032 negative queries across 90 tasks (Table R5).

Scoring and stability. For Base CoT and Custom CoT, we ignore answers that cannot be read as positive or negative. For each query with at least one valid answer, we calculate the fraction of valid answers that are correct. We average these fractions within each task, then average across tasks. GPT-5.2 Base CoT scored 59.64 ± 3.25% on the expanded V-LoL set (seeds 11, 23, 47). In paper Table 1, ± denotes the standard deviation across seeds.

Selection tolerance. We allow one binomial standard error with a one-image minimum; a second cutoff below 0.75 triggers full-pool reranking (Appendix B.2; response to Reviewer 3, Q2).

What does symbolic verification contribute? Executing a rule induced by Metagol reveals which negative support images it accepts and which positive ones it rejects. These counterexamples tell the MLLM which images to revisit. It rechecks the attributes and relations tested by the rule and revises the facts used in the next search. On CURI-90, the fraction of recorded attributes with a wrong value fell from 21.47% to 17.35% after reflection, even though Missing barely changed (7.07% to 7.00%, Figure 2). With the selector held fixed, enabling this feedback loop raised query accuracy from 86.81% to 92.19% (+5.38 points, Table 2). With the same executor and two-round reflection cap, Code CoT reached 77.68% using MLLM-written code to induce rules (Table R4).

Ground truth for Figure 2. The original Figure 2 caption defines Extra as a visually correct attribute irrelevant to the target rule. Section 4.4 describes in more detail how we apply this definition using the annotations for each object. If the rule uses only shape, a correct color is Extra and a misread color is Wrong. Both rates use all filled attributes in the image as their denominator (Table R7).

4. Downstream evidence: do ILP-CoT rules produce better images?

A successful image must preserve the learned rule in the new scene the user requests. If Direct Predict misses an interaction in the examples, its rule cannot tell the generator to preserve that interaction. With five positive and five negative examples, human evaluators judged 64% of ILP-CoT rules completely correct, compared with 14% for Direct Predict (paper Table 4). Figure 3 shows why this matters: both prompts place the family in the requested forest. Without the rule, the child walks between the adults; with the learned rule, the woman holds the child while the man looks on. The setting changes, but the family interaction is preserved.

Reviewer_mpTGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ILP-CoT combines MLLM perception with rule induction. The MLLM extracts visual facts and proposes rule structures. Metagol searches within these structures. Support errors guide revisions to both facts and proposals. A selector chooses the final program. The paper evaluates three classification benchmarks and a small image-customization application.

Critical strengths
  1. A useful training-free design. MLLM proposals reduce manual specification of the ILP search space. Shared object variables connect attributes and relations. Feedback can revise both visual facts and rule structures. The resulting programs can be executed and inspected.

  2. Strong empirical gains. ILP-CoT leads the prompting baselines across all four models and three classification benchmarks (Table 1). For Qwen3-VL, V-LoL accuracy rises from 58.97% with Custom CoT to 81.25%. Table 2 separately examines reflection and selection. Table 3 also compares alternative ILP backends.

Critical weaknesses
  1. The baselines do not isolate the benefit of ILP search. Appendix A states that Code CoT predicts through the MLLM; it does not execute the generated Prolog. ILP-CoT also adds feedback, repeated visual checks, candidate search, and selection. Custom CoT provides staged reasoning, but does not match these steps. The gains are convincing against the reported baselines. Their source remains unclear. A baseline that executes MLLM-generated programs with matched feedback and candidate budgets would address this concern.

  2. Key implementation and evaluation details are missing. Appendix B.2 retains programs with 'close' or 'similar' accuracy, but gives no numerical tolerances. The support split and subset-search schedule are also unspecified. These choices matter: the selector improves over support-argmax by 10.42 points on V-LoL and 10.08 on CURI-90 (Table 2). Classification support/query counts and variability across runs are not reported. The five repetitions in Appendix D concern customization, not the main classification results. These omissions limit reproducibility and confidence in smaller gains.

Questions for authors
  1. How does ILP-CoT compare with executed MLLM-generated rules under matched feedback, selection, and candidate budgets? Please also report inference cost.

  2. What are the exact selection tolerances, support splits, and subset-search settings? How were they chosen? Please report classification sample counts and variation across runs.

  3. How were Figure 2's fact errors annotated and normalized? Metagol checks consistency with extracted facts, not their visual truth. Please qualify the verification claim and distinguish irrelevant but correct facts from hallucinations.

1 comment

We thank the reviewer for the detailed feedback and for recognizing the gains over the reported baselines.

Q1. How do execution, feedback and search budgets compare?

Original Code CoT. The original prompt already required the MLLM to write Prolog from its visual observations and run the code in SWI-Prolog before returning a prediction. This baseline scored 47.66% on CLEVR-Hans with Qwen3-VL-32B (original Table 1).

Revised Code CoT. We remove the Metagol search stage from ILP-CoT. Given the same extracted facts, relations and labeled examples, the MLLM writes SWI-Prolog code that calls an ILP learner to induce candidate rules. In the full method, Metagol fills in the predicates of MLLM-proposed clauses. Both methods use the same rule selector, SWI-Prolog executor and two-round reflection cap.

Search budget and accuracy. On CLEVR-Hans, Code CoT evaluated 10.33 distinct rules per task and ILP-CoT evaluated 3,267. The 25-second limit applies per symbolic search call. With Qwen3-VL-32B, ILP-CoT led by 4.03 points on V-LoL, 18.23 on CLEVR-Hans and 14.51 on CURI-90. Table R4 reports time after fact extraction. Paper Table 5A counts successful MLLM calls across stages.

Table R4. Executed programs, Qwen3-VL-32B. Costs per task, mean ± SD (paper Table 1 and Figure 4A, C, E).

DatasetMethodAccuracy (%)Candidate-generation callsDistinct rules evaluatedRule-learning time (s)
V-LoLCode CoT79.17 ± 7.223.00 ± 0.0010.56 ± 0.7388.63 ± 14.80
ILP-CoT83.20 ± 3.202.89 ± 0.78622 ± 1841,089.51 ± 327.67
CLEVR-HansCode CoT68.49 ± 7.083.00 ± 0.0010.33 ± 0.5047.37 ± 7.62
ILP-CoT86.72 ± 0.523.33 ± 0.503,267 ± 25272.92 ± 50.38
CURI-90Code CoT77.68 ± 1.293.00 ± 0.009.24 ± 1.5561.50 ± 7.93
ILP-CoT92.19 ± 2.681.67 ± 0.492,690 ± 3801,013.50 ± 187.31

Q2. How were support sets split, rules selected and variability measured?

Splits and subset schedule. Early subsets balance classes; the final search uses the full fitting set. Selection images are excluded from initial fitting but may inform later reflection. Table R5 gives split and query counts and the schedule (§3.4; Appendix B.1).

Table R5. Support and queries (§4.1, paper Tables 6–7). Pairs give positive/negative counts. Each round ends with the full fitting set. Later rounds reshuffle images.

DatasetFitting subsets in search orderSelectionQueries/taskTask–query pairs/run
V-LoL(2,2) → (3,3) → (4,4) → (6,6) → (8,8)2/2128/128768
CLEVR-Hans(1,1) → (4,4) → (16,16)16/1664/64384
CURI-90(1,1) → (3,3) → (3,18)2/23–6/3–121,333

Tolerance and rationale. On full support, then on selection, retain scores within max⁡(1/n,p(1−p)/n)\max(1/n,\sqrt{p(1-p)/n}) of the best accuracy p on n images: one standard error, with a one-image minimum. We use 0.75 because, with two positive and two negative images, 3/4 is the lowest score that beats predicting the same class every time. We favor shorter rules if the cutoff after tolerance reaches 0.75. Otherwise, we rerank the full pool using class-balanced criteria (Appendix B.2).

Variation across runs. Table R6 reports mean ± SD over seeds 11, 23 and 47. With reflection the full selector leads on all three datasets; without it, support-argmax leads on two.

Table R6. Selector ablation with Qwen3-VL-32B. Query accuracy in %, mean ± SD (paper Table 2).

DatasetWith reflection Full selectorWith reflection Support-argmaxNo reflection Full selectorNo reflection Support-argmax
V-LoL83.20 ± 3.2056.51 ± 1.8055.08 ± 1.1754.43 ± 1.13
CLEVR-Hans86.72 ± 0.5280.47 ± 1.3077.60 ± 0.0080.47 ± 1.45
CURI-9092.19 ± 2.6886.30 ± 0.2686.81 ± 0.6786.91 ± 0.35

Q3. What counts as a fact error, and what does Metagol verify?

Verification. Metagol flags support-label disagreements, not visual falsehoods. The MLLM rechecks the image.

Annotation and normalization. We compared each extracted attribute with its ground-truth value for the same object in the dataset annotations. Per image, Missing counts unfilled rule-relevant attributes over all expected relevant attributes. Wrong and Extra use all filled attributes: an incorrect value is Wrong; a correct but irrelevant value is Extra, not a hallucination. CURI-90 Missing barely changed (Table R7; §4.4).

Table R7. Visual fact checks, Qwen3-VL-32B. Mean rates (%) before → after reflection (paper Figure 2).

DatasetMissingWrongExtra
V-LoL17.45 → 10.2529.66 → 19.8628.65 → 22.53
CLEVR-Hans34.30 → 28.3318.32 → 5.2268.73 → 62.73
CURI-907.07 → 7.0021.47 → 17.3552.70 → 49.77
Reviewer_EGGiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a neuro-symbolic method that combines MLLMs with ILP to perform faithful inductive reasoning on visual problems. It leverages the MLLM's perception to capture tokens/facts from the images and to propose an initial candidate rule. This rule is abstracted into a meta-rule used to search for fitting logic programs. If no fitting program is found in the first round, the MLLM is used to obtain feedback and improve both facts and candidate rules. The method proofs to be effective on visual inductive tasks with challenging rules.

Critical strengths

The method proposes an interesting combination of MLLMs and ILP that leverages the strengths of each and proves efficient, even on visually challenging datasets like V-Lol.

The text-to-image customized generation is an exciting application, and it is nice to see it implemented in the paper.

Critical weaknesses

The paper is vague on many methodological details; e.g., the retrieval of capture tokens in 3.2 and the failure reflection are very vague, especially since the feedback method is described mostly in prose. Further, no prompt details are provided and no code. It is unclear how much domain knowledge the MLLMs receive or if they are proposing the facts without any context.

Related work by Wüst et al. (2026) is missing. Their method uses MLLMs for visual rule induction as well, e.g., also on CLEVR-Hans and real-world images. While the methods differ in several aspects, the performance should ideally be compared, or at least a reasonable argument made why not.

If the above two weaknesses are addressed, I think this is a very good paper and I would be happy to raise my score.

--

Wüst, Antonia, et al. "Synthesizing visual concepts as vision-language programs." CVPR. 2026.

Questions for authors
  • Q1: It is not clear to me how much prior knowledge the MLLM receives in the capture token prompt. Do you assume domain knowledge here? Also, does the MLLM see all images at once or each one individually?
  • Q2: Could you provide more details on the reflection failure reflection part? How many prompts are exactly happening there, or is everything happening in one prompt?

Sorry for the brevity, and if I missed something; this is an emergency review.

Remark:

  • CLEVR-Hans was introduced in Stammer et al. (2021), not in Shindo et al. (2024) (line 307)

--

Stammer, Wolfgang, Patrick Schramowski, and Kristian Kersting. "Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations." CVPR, 2021.

1 comment

We thank the reviewer for taking on this review at short notice. We appreciate the encouraging assessment of our work and the opportunity to explain the method in greater detail.

Q1. What prior knowledge is supplied, and which images are shown together?

Capture tokens. We supply no task-specific domain knowledge in the capture-token prompt. The prompt asks the model to use its own visual knowledge to name visible objects, properties, and relations and to describe how to recognize them. It does not supply the target rule. The model builds a dictionary for each task from unlabeled support images and revises it as it sees more examples.

Image input. The model records facts one image at a time. When proposing rules, it compares the facts from labeled positive and negative fitting examples and receives predicate definitions, argument types, relation directions and the allowed Prolog syntax (§3.2 and Appendix E). Thus, recording an image and comparing examples to propose a rule are separate steps.

We have included more detailed illustrations of the prompts used in these major steps (Appendix E). We will also release all code and data upon formal release for reproducibility.

Q2. Is reflection a single prompt? What requires another call?

Each reflection round begins with one request containing selected support images, their labels and extracted facts, candidate rules, prediction errors and a summary of the fitting subsets searched. The same reply can suggest a clause revision and request a visual check. Each requested image check takes a separate call. We run at most two reflection rounds. Appendix E gives representative prompts. We plan to release the code and data with the paper.

Related work. How does VLP perform on CLEVR-Hans?

We followed the published VLP setup (Wüst et al., 2026), using the same MLLM as ILP-CoT for visual extraction and no additional perception training. ILP-CoT had higher mean accuracy for all four base models (Table R3; paper Table 1 and Appendix A).

Table R3. VLP and ILP-CoT on CLEVR-Hans. Query accuracy in %, mean ± SD (paper Table 1).

Base modelVLPILP-CoT
Qwen3-VL-32B64.41 ± 1.9286.72 ± 0.52
MiMo-V2.6-Pro80.90 ± 3.7288.72 ± 1.67
GPT-4o65.45 ± 3.3081.68 ± 2.53
GPT-5.266.23 ± 4.0790.71 ± 1.23

Dataset attribution. Section 4.1 now credits Stammer et al. (2021) for CLEVR-Hans.

Reviewer_pJr8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript proposes ILP-CoT, a neuro-symbolic framework that combines multimodal large language models (MLLMs) and inductive logic programming (ILP) for visual logical rule induction. MLLMs offer powerful perception ability, identifying logical facts from images, involving identifying entities and relations. ILP can produces interpretable and verifiable logical rules according to the logical facts. Experiments show the effectiveness of the proposed method. The authors also explore a potential application of ILP-CoT, i.e., text-to-image customized generation, using the learned rules to guide image generation.

Critical strengths

None

Critical weaknesses
  • Visual rule induction is not new. α\alpha ILP [1] and NEUMANN [2] (and related works) have already proposed entire visual ILP models. This work should be compared with them. If I did not ignore something, this work even does not mention that line of work.

  • Table 3: The comparison among ILP backends is not fair. The proposed method defines inductive bias through meta-rules for Metagol, but does not provide inductive bias for other ILP alternatives (at least not documented in the manuscript). Inductive bias is crucial for symbolic ILP approaches. To conduct a fair comparison, I suggest providing inductive bias for other ILP backend for a fair comparison.

[1] α\alpha ILP: thinking visual scenes as differentiable logic programs. Machine Learning 2023. [2] Learning differentiable logic programs for abstract visual reasoning. Machine Learning 2024.

Questions for authors

See Weaknesses.

1 comment

We would like to thank the reviewer for the detailed comments and suggestions. They helped us place ILP-CoT in the context of prior work on visual ILP and explain the inductive bias used by each backend.

1. How does ILP-CoT compare with visual ILP methods?

The original manuscript cited both works in the Introduction (Shindo et al., 2023, 2024). We now compare αILP and NEUMANN with ILP-CoT on the same support and query images, using the same base MLLM for visual extraction without additional perception training. ILP-CoT had higher mean query accuracy than both methods in all 12 model–dataset pairs (Table R1; paper Table 1).

The methods learn different things from the extracted facts. NEUMANN learns weights for candidate clauses by gradient descent (Shindo et al., 2024). In ILP-CoT, Metagol searches for predicate assignments in clause structures proposed by the MLLM; errors on support images guide later revisions.

Table R1. Visual ILP baselines. Query accuracy in %, mean ± SD across seeds 11, 23 and 47 (paper Table 1).

DatasetBase modelNEUMANNαILPILP-CoT
V-LoLQwen3-VL-32B53.91 ± 1.4554.34 ± 0.7283.20 ± 3.20
MiMo-V2.6-Pro52.26 ± 1.1153.04 ± 1.6274.35 ± 6.23
GPT-4o52.56 ± 1.2852.21 ± 0.6971.74 ± 5.02
GPT-5.252.78 ± 1.8751.87 ± 0.2788.28 ± 1.70
CLEVR-HansQwen3-VL-32B61.37 ± 6.9075.78 ± 1.3586.72 ± 0.52
MiMo-V2.6-Pro55.82 ± 3.5276.30 ± 0.0088.72 ± 1.67
GPT-4o53.21 ± 0.8070.83 ± 0.2681.68 ± 2.53
GPT-5.253.47 ± 1.8871.09 ± 0.0090.71 ± 1.23
CURI-90Qwen3-VL-32B69.26 ± 1.0965.95 ± 0.3092.19 ± 2.68
MiMo-V2.6-Pro58.40 ± 1.9370.11 ± 0.2294.19 ± 2.20
GPT-4o58.07 ± 3.7469.64 ± 0.0888.19 ± 3.50
GPT-5.253.88 ± 2.3467.40 ± 0.5196.22 ± 1.20

2. What inductive bias and search budget did each backend receive?

All four backends received the same visual facts and proposed rule patterns, with a 25-second limit per search call. For Aleph, ILASP, and XHAIL, we translate each MLLM proposal into background rules that preserve its combinations of conditions, object bindings, relations, and any quantifiers, negation, or numerical comparisons it contains. For example, “the image contains a red cube” preserves both the existence requirement and the requirement that redness and cube shape refer to the same object. The head mode specifies image classification as the learning target, while the body mode makes the proposal-derived conditions available for constructing classification rules. Each backend learns from the same visual facts, object relations, and labeled support examples, using the shared proposals from the initial and subsequent reflection rounds.

Table R2 compares query accuracy across all four backends. Each backend received the same proposed rule patterns in the form it supports. Aleph, ILASP, and XHAIL also used the same labeled examples, a shared candidate pool for each task, and the same final rule selector. Figure 4D of the paper reports their runtimes on that shared pool.

Table R2. Backend query accuracy with Qwen3-VL-32B, % (mean ± SD, paper Table 3).

DatasetMetagolAlephILASPXHAIL
V-LoL83.20 ± 3.2054.17 ± 2.15No rule outputNo rule output
CLEVR-Hans86.72 ± 0.5267.71 ± 0.0033.07 ± 0.0033.07 ± 0.00
CURI-9092.19 ± 2.6865.33 ± 3.2465.50 ± 3.6465.55 ± 3.05

“No rule output” means the backend did not consistently induce a rule on that dataset.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.