- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- ILP-CoT combines MLLM-based visual grounding and rule proposals with Metagol-based program search. Failure reflection revises the visual facts and proposed structures, while a support-based selector chooses the final executable rule. The core induction pipeline does not require additional model training.
- The paper evaluates visual classification on V-LoL, CLEVR-Hans, and CURI-90 with four base MLLMs. It also introduces a small customization dataset and demonstrates how inferred rules can guide image generation.
Critical strengths
- The division of responsibilities is well motivated. The MLLM handles perception and proposes plausible structures, while the symbolic system searches for and executes concrete programs. This yields inspectable intermediate facts and rules.
- The pipeline improves over prompting baselines across backbones. ILP-CoT outperforms the three prompting baselines in all twelve model and dataset combinations and reaches 81.25%, 82.55%, and 92.19% with Qwen3-VL-32B. The size of this margin should be read with care because the baselines are close to chance on two datasets (Weakness 4).
- The component analyses are useful. Table 2 examines both reflection and program selection rather than only the full system. Their effects differ across datasets, which suggests that they address distinct sources of error.
Critical weaknesses
-
Comparisons with closely related methods are missing.
- Table 1 compares prompting strategies, and Tables 2 and 3 examine internal components and ILP backends. These controls support the authors' design choices but do not place the method in the literature.
- Vision-Language Programs and Pix2Code are the most relevant candidates. Both have public code, with CLEVR-Hans support in VLP and CURI evaluation in Pix2Code. Published numbers should not be copied without accounting for differences in support size, metric, perception training, and task split.
- NEUMANN (Shindo et al., 2024) is cited only in passing. A brief distinction would clarify the novelty.
-
The baselines do not isolate the benefit of ILP search.
- According to Appendix A, Code CoT produces both the Prolog rule and the final prediction through the MLLM. The comparison therefore changes both how rules are found and how they are applied.
- Hypothesis Search (Wang et al., 2024) and hypothesis refinement (Qiu et al., 2024) already let an LLM propose programs that an interpreter executes and refines. A baseline of this kind with the same MLLM and the same extracted facts would identify the added value of Metagol. Model calls, candidate counts, and runtime should also be reported.
-
The generation results do not yet show an advantage from ILP-based induction.
- Table 4 evaluates rule descriptions, not generated images. The generation examples only compare prompts with and without the learned rule.
- A fair test would fix the generator and customization and compare Direct Predict rules with ILP-CoT rules. Otherwise the claims should be limited to an illustrative application.
-
Baseline strength and evaluation scale are unclear.
- V-LoL and CLEVR-Hans are binary tasks, yet all 24 baseline results on them lie between 40.22% and 58.97%, and 9 are below 50%. GPT-5.2 with Base CoT reaches only 51.18% on V-LoL. This raises the question of whether the baselines are set up fairly.
- Each of these two datasets has only three tasks (Section 4.1), with no repeated runs or variance. Most ILP-CoT results on V-LoL (81.25, 70.83, 62.50, 58.33, 54.17) are multiples of 1/48, which suggests about 48 queries in total. A 10-point gap would then correspond to about five images.
- Support and query counts, seeds, and the number of runs should be stated. The selector tolerance for "close" and "similar" accuracy is also unspecified (Appendix B), although the selector drives large gains in Table 2.
-
The fact-correction analysis does not support the formal-verification interpretation.
- Fig. 2 gives no protocol for obtaining ground-truth facts. Its "Extra" category also counts visually correct but irrelevant facts as errors.
- The text claims that errors consistently decrease, yet CURI Missing stays at 1.15%. The corrections also come from MLLM rechecks during reflection, not from ILP verification.
Questions for authors
- Comparison with existing methods
- Could the authors compare with VLP on CLEVR-Hans and, where compatible, Pix2Code on CURI-90 under identical support and query examples? Differences in supervision or compute should be reported.
- Attribution of the improvement
- How does ILP-CoT compare with MLLM-generated rules executed by an external interpreter on the same facts, with comparable candidate and feedback budgets?
- What are the model-call and runtime costs of each method?
- Evaluation reliability
- How are unparsable baseline answers handled, and are the query sets class-balanced?
- How many queries does each task contain, and how stable are the results across runs?
- What tolerances does the selector use, and how were the ground-truth facts in Fig. 2 obtained?
- Downstream evidence
- Does an ILP-CoT rule improve the generated image over a directly inferred rule under the same generation setup?
Limitations
- Consistency is not a guarantee of the intended rule. A program may fit the extracted facts and support labels through an unintended correlation. Formal verification claims should be limited to consistency with the supplied representation and examples.
- The generation application is preliminary. It works as a demonstration but does not establish superiority in relation-aware generation.
- The pipeline is promising, but the evaluation relies on prompting controls that are close to chance and on internal substitutions. The added value over existing approaches is therefore hard to judge.
- I would raise my score if fair comparisons with closely related methods show a meaningful benefit and the evaluation protocol is made explicit. I do not expect an exhaustive benchmark suite or superiority in every setting. The priority is evidence that the integration helps beyond stronger prompting, external execution, or additional search.
2 comments
4. Downstream evidence: do ILP-CoT rules produce better images?
A successful image must preserve the learned rule in the new scene the user requests. If Direct Predict misses an interaction in the examples, its rule cannot tell the generator to preserve that interaction. With five positive and five negative examples, human evaluators judged 64% of ILP-CoT rules completely correct, compared with 14% for Direct Predict (paper Table 4). Figure 3 shows why this matters: both prompts place the family in the requested forest. Without the rule, the child walks between the adults; with the learned rule, the woman holds the child while the man looks on. The setting changes, but the family interaction is preserved.
We would like to thank the reviewer for the detailed comments and suggestions. We appreciate your recognition that our control experiments support our design choices.
1. Comparison with existing visual rule learners
VLP and Pix2Code. Both used ILP-CoT's support and query images and the same visual extractor, with no additional perception training. With Qwen3-VL-32B, VLP reached 64.41% versus 86.72% on CLEVR-Hans; Pix2Code reached a preliminary 68.00% versus 92.19% on CURI-90 (Table R8). Table R3 covers all four VLP base models.
NEUMANN. It learns clause weights; Metagol searches for predicate assignments. Table R1 compares both approaches and αILP.
Table R8. Requested baselines with Qwen3-VL-32B. Query accuracy in % (paper Table 1).
± denotes the standard deviation across seeds. Dashes indicate ongoing results. Pix2Code induced no rule on V-LoL or CLEVR-Hans. Its CURI-90 score is preliminary, with further runs pending.
2. Attribution of the improvement and computational cost
What is matched? Hypothesis Search and hypothesis refinement execute proposals on the same initial facts and examples, with the same reflection cap and scoring (Appendix A; Table R8). The original Code CoT prompt required SWI-Prolog execution. Revised Code CoT replaces Metagol search with MLLM-written code. Both methods select rules on the same support examples and execute them on the same query facts. On CLEVR-Hans, Code CoT evaluated 10.33 rules per task and reached 68.49% accuracy. ILP-CoT evaluated 3,267 and reached 86.72% (Qwen3-VL-32B, Table R4).
Calls and runtime. Table R9 compares successful MLLM calls and rule-learning time for all four methods in paper Table 5A. Table R4 separately counts calls used to generate candidates.
Table R9. Computational cost with Qwen3-VL-32B, mean per task (paper Table 5A).
Calls = successful MLLM calls; minutes = rule-learning time after fact extraction. Paper Table 5B counts each shared request and its tokens once across ILP-CoT, Code CoT, NEUMANN and αILP.
3. Evaluation reliability
Query counts and class balance. V-LoL now has 128 positive and 128 negative queries per rule, replacing the original 48-query evaluation. CLEVR-Hans has 64 of each per concept (384 pairs; 192 distinct images). CURI-90 has 301 positive and 1,032 negative queries across 90 tasks (Table R5).
Scoring and stability. For Base CoT and Custom CoT, we ignore answers that cannot be read as positive or negative. For each query with at least one valid answer, we calculate the fraction of valid answers that are correct. We average these fractions within each task, then average across tasks. GPT-5.2 Base CoT scored 59.64 ± 3.25% on the expanded V-LoL set (seeds 11, 23, 47). In paper Table 1, ± denotes the standard deviation across seeds.
Selection tolerance. We allow one binomial standard error with a one-image minimum; a second cutoff below 0.75 triggers full-pool reranking (Appendix B.2; response to Reviewer 3, Q2).
What does symbolic verification contribute? Executing a rule induced by Metagol reveals which negative support images it accepts and which positive ones it rejects. These counterexamples tell the MLLM which images to revisit. It rechecks the attributes and relations tested by the rule and revises the facts used in the next search. On CURI-90, the fraction of recorded attributes with a wrong value fell from 21.47% to 17.35% after reflection, even though Missing barely changed (7.07% to 7.00%, Figure 2). With the selector held fixed, enabling this feedback loop raised query accuracy from 86.81% to 92.19% (+5.38 points, Table 2). With the same executor and two-round reflection cap, Code CoT reached 77.68% using MLLM-written code to induce rules (Table R4).
Ground truth for Figure 2. The original Figure 2 caption defines Extra as a visually correct attribute irrelevant to the target rule. Section 4.4 describes in more detail how we apply this definition using the annotations for each object. If the rule uses only shape, a correct color is Extra and a misread color is Wrong. Both rates use all filled attributes in the image as their denominator (Table R7).