acceptodds
Under review as a conference paper at ICLR 2027

RIPA: Rule Induction from Probabilistic Atoms for Closed-Loop Multimodal Inductive Reasoning

Abstract

Multimodal Large Language Models (MLLMs) have made substantial progress in visual understanding, yet remain less reliable in visual inductive reasoning and provide limited rule-level interpretability. Neuro-symbolic methods, by contrast, support explicit rule induction and traceable inference, but their integration with MLLMs for rule-guided prediction and image-specific rule verification remains insufficiently explored. In this paper, we propose RIPA, a closed-loop framework integrating neuro-symbolic rule induction with MLLM reasoning. RIPA first uses an MLLM to extract probabilistic atoms describing object attributes, relations, and overall scene structure from images. A Template-Driven Differentiable Rule Learner (TDRL) then induces sparse class-specific rules from these atoms and stores selected rules in a rule bank. During inference, TDRL scores candidate classes and retrieves their associated rules. These scores and rules, together with the image and probabilistic atoms, are fed back to the MLLM for prediction and image-specific rule verification. Finally, a consistency-aware module fuses the TDRL scores and MLLM outputs using rule agreement and conflict signals to produce the final prediction, thereby closing the reasoning loop. Experiments on two established object-centric visual reasoning benchmarks and our RAVEN-Custom benchmark show that RIPA improves over the neuro-symbolic MLLM baseline ILP-CoT by 10% on average. Moreover, RIPA exports explicit symbolic rules and rule-level evidence, making its predictions interpretable and traceable.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neuro-Symbolic Reasoning, Visual Inductive Reasoning, Multimodal Large Language Models, Differentiable Rule Learning, Interpretable Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_pc38OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RIPA combines MLLM perception with differentiable rule induction. It extracts probabilistic atoms, learns class-specific rules with TDRL, and asks an MLLM to verify retrieved rules. A fusion module combines class scores with rule agreement and conflict. Experiments cover CLEVR-Hans, Kandinsky Patterns, and a fixed-rule binary variant of RAVEN. The paper reports improved classification and exports readable rules.

Critical strengths
  1. The integration is well motivated. Retaining uncertain atoms avoids early hard decisions. Explicit rule retrieval and verification provide inspectable intermediate evidence.

  2. The evaluation includes three MLLM backbones, hard-atom and TDRL-only ablations, and oracle-atom analysis. The full system improves over TDRL alone across the reported datasets. The supplementary code and dataset generators also help reproducibility.

Critical weaknesses
  1. Task-specific rule knowledge confounds the induction claim. The supplied Hans7 configuration enables init_hans7_skeleton(). This initializes class-specific rule structures and ground-truth count thresholds, including 2, 5, and 3 (chirp/models/tdrl.py). Kandinsky initialization also presets task-specific predicates and attributes. These parameters remain learnable, but the initialization supplies substantial prior knowledge. Separately, _kandinsky_task_guidance() inserts each task's positive-pattern definition into the verification prompt (chirp/reasoning/prompt_builder.py). For example, it specifies the red-triangle rule and its required nearby object. The paper does not clearly disclose these inputs or isolate their contribution. Please clarify whether the reported runs used these defaults. Without controls, the gains cannot be attributed cleanly to learned rules and verification.

  2. The comparison with ILP-CoT uses unequal supervision. Appendix C gives ILP-CoT 20 support images per class. TDRL uses the full training split, including 2,000 images per Kandinsky task and RAVEN configuration. Thus, on these binary tasks, the comparison uses 40 versus 2,000 labeled training images. A matched-data comparison is needed to separate methodological gains from additional supervision.

Questions for authors
  1. Which reported runs used the class-specific initialization and hand-written rule guidance? Please provide results without them and baselines given the same rule knowledge.

  2. How does RIPA compare with ILP-CoT under identical labeled support budgets? A learning curve would help assess the contribution of extra data.

  3. How faithfully do the exported rules reproduce the soft model's predictions? Figure 8 omits size and color conditions in the 2x2 and 3x3 exports. Please report executable-rule accuracy and agreement with TDRL and final predictions.

1 comment

We sincerely thank you for your careful review of our work and for your questions.

Q1 and W1

Which reported runs used the class-specific initialization and hand-written rule guidance? Please provide results without them and baselines given the same rule knowledge.

Hans3 used ground-truth-guided initialization and ground-truth rule checking. Hans7 used a class-specific skeleton with count thresholds 2, 5, and 3. Kandinsky used task-specific TDRL initialization and written positive-class definitions in verifier prompts. For example, the red-triangle prompt specifies a red triangle near an object that is neither red nor triangular.

Additional guidance-removal experiments. Removing Hans3 rule guidance gave 87.69% test accuracy. On Hans7, three paired seeds gave 84.95% mean accuracy with the skeleton and 87.52% without it. All runs without the skeleton learned that class 3 contains small brown, green, and purple objects, but did not export the additional requirement of at least five small objects. Removing the written Kandinsky target definition gave 87.81% mean accuracy across four tasks and 1,600 test images.

Additional target-rule baseline experiments. We also supplied target-rule information to baselines. Qwen3-VL-8B-Instruct received the target definition, image, and atom table, but no training examples; it classified 931/1,600 correctly. ILP-CoT received the same 200 support images per task as RIPA, with target clauses added to its candidates. It classified 986/1,600 correctly. Removing those clauses while retaining task-specific predicates changed neither the selected rules nor any prediction. These controls show that performance remains after the tested guidance is removed, while supplying explicit target clauses did not change ILP-CoT’s result.

Q2 and W2

How does RIPA compare with ILP-CoT under identical labeled support budgets? A learning curve would help assess the contribution of extra data.

RIPA retains its advantage when both methods use identical labeled support sets. Additional matched-budget learning-curve experiment. We compared both methods using the same nested, class-balanced subsets at five budgets, three seeds, and the same 400 test images per Kandinsky task. TDRL was retrained from its task-specific initialization; ILP-CoT retained its task-specific predicates and seed clauses (Appendix F, Table 6, Figure 9).

Labeled images per task401002001,0002,000
RIPA (%)90.5890.5291.1091.0090.92
ILP-CoT (%)60.6060.7959.9661.6361.63

RIPA achieved higher mean accuracy on all four tasks at every budget. Its advantage at 40 images per task was 29.98 points. Increasing the budget from 40 to 2,000 raised its mean accuracy by only 0.34 points. Thus, the gap remains when the labeled-data budget is matched.

Q3

How faithfully do the exported rules reproduce the soft model’s predictions? Figure 8 omits size and color conditions in the 2x2 and 3x3 exports. Please report executable-rule accuracy and agreement with TDRL and final predictions.

The exported rules reproduce most model decisions, but this does not establish that they recover all target conditions. Additional executable-rule and counterfactual experiments. We executed the exported conjunctions using the most probable attribute values and compared them with labels, soft TDRL, and final RIPA predictions. The RAVEN 3×3 test split contained 400 rows; 47 had invalid or empty perception outputs. We evaluated the remaining 353 rows (178 positive, 175 negative).

RAVEN layoutRule accuracyAgreement with TDRLAgreement with final RIPA
2×275.00% (300/400)90.75% (363/400)91.00% (364/400)
3×368.27% (241/353)98.30% (347/353)97.45% (344/353)

On 3×3, the export reproduced 347/353 TDRL decisions, including errors. We then changed only the final panel’s size or color annotations in 378 valid rows, creating 756 invalid variants. The exports accepted all 756; TDRL accepted 709. Both methods accepted all 283 valid position-only controls. Thus, the exports are largely faithful to classifier behavior, including some errors, while the counterfactuals show that recovery of the full target concept remains incomplete.

Reviewer_vNqYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RIPA, a framework for visual inductive classification that combines MLLM perception with a trainable neuro-symbolic rule learner. It extracts probabilistic atoms from images, learns class-specific rules through TDRL, and exports them to a fixed rule bank. During inference, an MLLM predicts classes and assesses visual support for retrieved rules. Final predictions combine symbolic and MLLM scores with rule agreement and conflict signals. Experiments cover CLEVR-Hans, Kandinsky Patterns, and RAVEN-Custom.

Critical strengths
  • Comprehensive and coherent framework design. RIPA integrates probabilistic perception, differentiable rule induction, visual verification, and prediction fusion into a unified pipeline. The explicit rules and intermediate evidence make the system’s reasoning process more accessible than final predictions or free-form explanations alone.

  • Controlled evidence for the combined inference stage. With the trained TDRL and probabilistic atoms held fixed, Table 3 reports improvements of 5.00–10.71 percentage points across the four benchmark groups after adding MLLM reasoning and fusion. This supports the usefulness of the additional stage as a whole, although the contributions of its individual mechanisms remain unresolved.

Critical weaknesses
  • Unequal supervision budgets. According to Appendices C and D, prompting baselines and ILP-CoT use 20 support images per class, whereas RIPA trains on 2,000 examples per Kandinsky task or RAVEN-Custom configuration. Without matched-data comparisons, the reported gains cannot be confidently attributed to the proposed design rather than the larger supervision budget.

  • Incremental design and insufficient isolation of the key contribution. ILP-CoT already combines MLLMs with symbolic rule induction and uses learned rules to guide MLLM classification. In addition to probabilistic rule learning, RIPA introduces explicit rule-support scoring and consistency-based fusion. However, the ablations do not isolate the contributions of these mechanisms.

  • Insufficient quantitative validation of exported rules. Rule interpretability is a central contribution, but its evaluation primarily relies on qualitative examples. The paper does not sufficiently quantify the correctness of exported rules, their faithfulness to TDRL, or how rule errors affect final predictions.

  • Limited scope of closed-loop feedback. TDRL and the rule bank remain fixed during inference. MLLM verification adjusts final class scores but does not revise the induced rules. The feedback can therefore penalize unsupported rule activations but cannot repair missing or incorrect rule conditions. This limits the feedback to decision-level correction rather than adaptive rule refinement.

Questions for authors
  1. Matched supervision budgets. How does RIPA compare with ILP-CoT when both methods use the same labeled examples for rule induction? Are the reported gains maintained when RIPA is restricted to the baseline’s support budget?

  2. Rule correctness and faithfulness. What classification accuracy is obtained by directly executing the exported rules, and how often do their predictions agree with TDRL? How does performance change on examples that violate conditions omitted from the learned rules?

Please also address the concerns about component ablations and the scope of closed-loop feedback raised in the weaknesses.

3 comments

We sincerely thank you for your careful review of our work and for your questions.

Q1 and W1

Matched supervision budgets. How does RIPA compare with ILP-CoT when both methods use the same labeled examples for rule induction? Are the reported gains maintained when RIPA is restricted to the baseline’s support budget?

Yes. RIPA retains its advantage with matched labeled examples. Additional matched-budget experiment. With Qwen3-VL-8B-Instruct, we compared five budgets using identical nested, class-balanced subsets and the same 400 test images per task across three seeds. TDRL was retrained from its task-specific initialization; ILP-CoT retained its task-specific predicates and seed clauses (Appendix F, Table 6, Figure 9).

Labeled images per task401002001,0002,000
RIPA (%)90.5890.5291.1091.0090.92
ILP-CoT (%)60.6060.7959.9661.6361.63

RIPA led on every task at every budget. Its advantage was 29.98 points at 40 images per task; increasing the budget to 2,000 changed mean accuracy by only 0.34 points. Thus, extra labeled data does not explain the advantage.

Q2 and W3

Rule correctness and faithfulness. What classification accuracy is obtained by directly executing the exported rules, and how often do their predictions agree with TDRL? How does performance change on examples that violate conditions omitted from the learned rules?

Exported rules reproduce most classifier decisions, but they do not recover every target condition. Additional executable-rule evaluation. We executed each conjunction on the most probable attribute values and compared it with labels, soft TDRL, and final RIPA. Of 400 RAVEN 3×3 test rows, 47 had invalid or empty perception outputs. We evaluated the remaining 353 (178 positive, 175 negative).

RAVEN layoutRule accuracyAgreement with TDRLAgreement with final RIPA
2×275.00% (300/400)90.75% (363/400)91.00% (364/400)
3×368.27% (241/353)98.30% (347/353)97.45% (344/353)

Additional counterfactual experiments. From 378 valid RAVEN rows, we created 756 variants violating omitted size or color conditions. The exports accepted all 756; TDRL accepted 709 and rejected 47. Both accepted all 283 valid position-only controls. On Hans3, changing the irrelevant cylinder color or cube material caused the export to reject all 40 scenes in each condition; TDRL remained correct on 39/40 and 36/40. Thus, exports can track classifier errors, while extra or missing conditions cause specific failures.

W2 — Component ablations

The ablations do not isolate the contributions of these mechanisms.

Table 3 measures the combined contribution of MLLM reasoning and fusion over TDRL alone. Additional component-isolation experiment. On 400 RAVEN 2×2 rows, we fixed images, atoms, rules, and TDRL. Each arm used one Qwen3-VL-8B-Instruct call per row. Direct rescoring, verification-derived class scores, unweighted rule support, and the agreement reward each matched TDRL (299/400). A conflict penalty selected on 400 validation rows corrected three errors and introduced one (301/400). Shuffling support scores across images reached at least 301/400 in 603/1,000 permutations. Thus, the small accuracy gain alone does not establish an image-specific verification benefit.

Additional rule-violation diagnostic. On 50 edited positive rows, the verifier identified 49/50 count violations and 48/50 type violations. In matched-budget Kandinsky, verification and fusion changed one of 24,000 predictions, correcting one TDRL error (Appendix F).

Verification provides interpretable rule-level evidence; fusion rewards supported activations and penalizes unsupported ones. The isolated accuracy gain is small in the RAVEN control.

W4 — Scope of closed-loop feedback

The feedback mechanism adjusts the final class scores while keeping TDRL and the rule bank fixed. Its role is therefore to correct the current decision, not to revise incomplete or incorrect rules.

Yes, this is the scope of RIPA’s closed loop. On the unmodified RAVEN 2×2 test, the conflict penalty corrected three errors and introduced one, increasing correct predictions from 299 to 301.

Additional feedback-scope experiment. We built 300 images from 50 positive originals, each with five variants changing only the last panel. Count, type, size, and color edits each produced 50 negatives; position-only edits produced 50 valid positives. Among 26 originals correctly classified as positive, count and type violations were rejected in 25 and 26 cases; omitted size and color violations were rejected in only 3 and 15 cases. With the validation-selected conflict weight, fusion and TDRL made identical predictions on all 300 images.

Feedback can affect the current decision, but cannot repair omitted conditions or update the rule bank.

Thank you for the detailed response and the additional experiments. The matched-budget results help address my concern regarding unequal supervision. However, the additional analyses reinforce the concern that the proposed closed-loop feedback remains primarily a decision-level correction mechanism, without the ability to refine incomplete or incorrect induced rules. Moreover, the relatively low accuracy of directly executed exported rules raises concerns about their faithfulness to the target concepts. Finally, despite the improvements over ILP-CoT, I remain unconvinced that the additional rule-support verification and consistency-based fusion provide sufficiently substantial novelty beyond the existing neuro-symbolic rule induction paradigm. Overall, while I appreciate the authors' efforts, these concerns remain insufficiently resolved. I therefore maintain my original score.

Thank you for acknowledging the matched-budget results and clarifying the remaining concerns. We address them with new Qwen3-VL-8B-Instruct experiments.

W4 — Refining the induced rule bank

However, the additional analyses reinforce the concern that the proposed closed-loop feedback remains primarily a decision-level correction mechanism, without the ability to refine incomplete or incorrect induced rules.

We have extended the feedback loop to update TDRL and its exported rules. We select these updates on validation data and keep the refined rule bank fixed during testing.

Additional rule-refinement experiments. On RAVEN, the verifier compares panels directly, without receiving Stage-1 atom text. We feed the verified size and color relations back into TDRL. A new condition is retained only if it improves validation accuracy. This recovers the omitted size and color conditions on both 2×2 and 3×3.

The updated exports reject 756/756 symbolic counterfactuals that violate size or color. The original exports accepted all 756. The updated rules also accept 283/283 valid position-only controls.

On CLEVR-Hans3, validation checks identify unsupported color literals and support the small metal cube condition. Removing the spurious literals raises class-1 executable-rule accuracy from 66.80% to 99.16%. On Hans7, refinement raises executable-rule accuracy on ground-truth scene atoms from 52.08% to 74.63%.

The feedback therefore changes the rules themselves. It adds missing conditions and removes unsupported conditions.

Q2 and W3 — Executable-rule accuracy and fidelity

Moreover, the relatively low accuracy of directly executed exported rules raises concerns about their faithfulness to the target concepts.

Rule refinement improves accuracy against the target labels. The exports also closely reproduce TDRL on Hans3 and Kandinsky.

Additional executable-rule evaluation. On ground-truth attributes, the refined RAVEN rules achieve 100% accuracy on both layouts. On attributes verified by the 8B model, they achieve 81.50% on 2×2 and 71.10% on 3×3.

On CLEVR-Hans3, refinement raises overall executable-rule accuracy from 65.69% to 98.27% across 2,250 test scenes. Agreement with soft TDRL rises from 65.64% to 97.42%.

On the four Kandinsky tasks, direct rule execution reaches 99.25% mean accuracy, with relation thresholds calibrated on validation data. The exports agree with soft TDRL and final RIPA on 98.75%–100.00% of test images.

These results establish accurate rule execution and high prediction agreement on Kandinsky and Hans3. The RAVEN counterfactual tests also show that the refined rules enforce the previously omitted conditions.

W2 — Contribution of rule-support verification and fusion

Finally, despite the improvements over ILP-CoT, I remain unconvinced that the additional rule-support verification and consistency-based fusion provide sufficiently substantial novelty beyond the existing neuro-symbolic rule induction paradigm.

The contribution is to connect image-specific rule checks to both prediction and rule refinement. RIPA checks individual conditions against the image and uses their support to reward agreement and penalize conflicts. The added validation feedback also uses these checks to refine the learner.

Additional component controls. On RAVEN, we compared clause verification with direct class rescoring using the same images, complete rule text, and one MLLM call per row. All fusion weights were selected on validation data. The first verification comparison keeps TDRL fixed.

Test accuracyRAVEN 2×2 (400)RAVEN 3×3 (353)
TDRL only74.75%67.14%
TDRL + direct rescoring74.75%67.14%
TDRL + rule verification and fusion79.25%69.69%
Refined TDRL + rule verification and fusion82.00%75.35%

The 3×3 comparison uses 353 rows with valid perception outputs. With TDRL fixed, verification adds 4.50 and 2.55 percentage points. Shuffling support scores across images gives mean accuracies of 65.87% and 66.11%. The unshuffled scores outperform all 1,000 permutations (p < 0.001).

A separate 700-image red-triangle control also improves accuracy from 83.86% to 88.57%, outperforming shuffled support scores (p = 0.007).

These controls isolate the value of image-specific rule verification beyond direct rescoring. Together with the rule-refinement results, they show how visual checks improve both final predictions and the rules used to produce them.

Reviewer_H9aJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RIPA learns class-specific rules from probabilistic visual atoms and uses them to guide MLLM prediction and image-specific verification. Specifically, TDRL learns class-specific rules and exports them to a fixed rule bank. The final prediction combines symbolic scores, MLLM scores, and rule agreement signals, and the experiments cover CLEVR-Hans, Kandinsky Patterns, and RAVEN-Custom, a fixed-rule, row-level binary classification task derived from RAVEN.

Critical strengths
  • The learned rules from the method have a concrete and significant role in inference. The method guides the MLLM and provides conditions to test against each image, so exposing both the rules and their support scores makes the intermediate reasoning easier to inspect. I think this approach delivers inspectable intermediate representations and a concrete mechanism for incorporating visual verification into symbolic predictions.
  • The TDRL-only ablation shows a clear benefit from the subsequent stages. With the trained TDRL, atoms, and rule bank held fixed, adding MLLM reasoning and fusion improves accuracy by 5.00~10.71 percentage points (Table 3). The hard-atom results also support retaining uncertainty in the visual representation, and this is additional evidence that the overall framework boosts classification performance.
Critical weaknesses
  • The labeled-data budgets differ substantially between RIPA and its main baselines. Appendix C.2 and C.4 use 20 support examples per class for the prompting baselines and ILP-CoT, while RIPA trains TDRL on the training split. On each Kandinsky task, this means 40 examples for the baselines versus 2,000 for RIPA. The authors report that larger support sets did not consistently help in preliminary validation, but provide no matched-budget results. I find it difficult to judge how much of the improvement over ILP-CoT comes from the proposed method and how much comes from the additional labeled examples. Table 3 supports the benefit of adding reasoning and fusion to TDRL, and it does not settle this comparison with earlier methods. Please clarify this point if I got misunderstanding.
  • Some exported rules miss conditions that define the target concept. In Figure 8, the rules for both the 2×2 and 3×3 RAVEN-Custom configurations omit the required size and color transformations. Figure 6 also contains extra attributes and changed relation labels in the CLEVR-Hans rules. I think these discrepancies matter because recovering explicit task rules is a main contribution. The authors acknowledge them but do not measure their effect. RAVEN-Custom uses fixed-rule classification of individual rows, which further limits what its accuracy tells us about learning other transformation combinations. The CLEVR-Hans test results offer useful evidence under distribution shift. However, I would still like to see whether the learned rules reject examples that violate an omitted condition and remain valid when irrelevant attributes change.
Questions for authors
  • How often does Top-K selection exclude the correct class, particularly with four candidates on CLEVR-Hans7? What happens to final accuracy when all classes are passed to verification?
  • How does inference cost compare with ILP-CoT in MLLM calls, token usage, and latency, excluding one-time rule induction and caching?
3 comments

We sincerely thank you for your careful review of our work and for your questions.

W1 — Unequal supervision budgets

The labeled-data budgets differ substantially between RIPA and its main baselines.

The original comparison did not match labeled-data access. Prompting baselines used 20 support images per class; Kandinsky ILP-CoT sampled from 100 positives and 100 negatives per task, while TDRL used all 2,000 training images. The statement about larger support sets referred only to prompting baselines.

Additional matched-budget experiment. We used Qwen3-VL-8B-Instruct to compare the methods under matched budgets. Both methods used identical nested, class-balanced subsets and the same 400 test images per task. TDRL was retrained from its task-specific initialization at each budget; ILP-CoT retained its task-specific predicates and seed clauses. The table reports means over four tasks and three seeds (Appendix F).

Labeled images per task401002001,0002,000
RIPA (%)90.5890.5291.1091.0090.92
ILP-CoT (%)60.6060.7959.9661.6361.63

RIPA led on every task at every matched budget. Its mean advantage at 40 images per task was 29.98 percentage points. Increasing its budget from 40 to 2,000 images raised mean accuracy by only 0.34 points. The matched-budget experiment therefore confirms that the advantage remains even at the baseline’s support budget.

W2 — Missing and extra conditions in exported rules

The exported RAVEN rules omit required size and color conditions, while some CLEVR-Hans exports include irrelevant attributes. The key question is whether these omissions and additions affect decisions when the relevant or irrelevant attributes are changed.

The exported rules do not fully recover the target concept. Additional Hans3 counterfactual experiment. On Hans3, the class-0 export requires a yellow cylinder and metal cube, while the target concept requires a large cylinder and large cube, regardless of color or material. We edited 40 class-0 scenes accepted by the export:

EditExport retains class 0Soft TDRL retains class 0
Cylinder: yellow → gray0/4039/40
Cube: metal → rubber0/4036/40
Distractor color only40/4040/40

The extra attributes made the export more restrictive than TDRL.

Additional RAVEN counterfactual experiment. We created 756 variants from 378 valid rows by changing omitted size or color conditions. The exports accepted all 756 invalid variants; TDRL accepted 709 and rejected 47 (all color edits in 3×3). Both methods accepted all 283 controls in which only position changed and the target rule remained valid.

These tests quantify both failure modes: omitted conditions can go undetected, while extra attributes can make an exported rule overly restrictive.

Q1

How often does Top-K selection exclude the correct class, particularly with four candidates on CLEVR-Hans7? What happens to final accuracy when all classes are passed to verification?

The correct class is rarely excluded by Top-4 on Hans7. Additional candidate-coverage experiment. With K=4, the correct class was included for 5,226/5,250 Hans7 images (99.54%). Passing all seven classes to the same Qwen3-VL-8B-Instruct verifier made the correct class available in the remaining 24 cases, but did not change any prediction. Both settings achieved 4,333/5,250 (82.53%). Seven candidates increased verification tokens by 18.2%. Thus, Top-4 was sufficient in this test.

Q2

How does inference cost compare with ILP-CoT in MLLM calls, token usage, and latency, excluding one-time rule induction and caching?

RIPA makes fewer MLLM calls, but it does not have lower token usage or latency in this comparison. Additional inference-cost experiment. We timed local Qwen3-VL-8B-Instruct inference on the same 80 held-out Kandinsky images. Per-image means exclude model loading, warm-up, rule induction, and RIPA’s atom-cache construction.

MethodMLLM callsInput tokensOutput tokensLatency
RIPA (cached atoms + image)11,687.0483.511.10 s
ILP-CoT (raw image)21,292.2553.980.77 s

RIPA used fewer calls, but not fewer tokens or lower latency.

I appreciate the authors' clarification. However, my main remaining concern is how missing conditions in the exported rules affect the final predictions. It is unclear whether MLLM verifier can correct these errors when these omissions are explicitly provided.

From my understanding, in the RAVEN 2x2 experiment, among the 26 examples that RIPA originally classifies correctly as positive, only 3 size-violating and 15 color-violating variants were correctly rejected. In other words, a large amount of the samples remains positive after a size or a color violation. I think this raises concerns about the reliability of rule-grounded verification, so I suggest the authors to do follow-up controlled experiments by only the omitted size condition to the rule provided to the verifier while keeping other variables unchanged.

Additionally, I am not quite sure about the 47 RAVEN 3x3 examples excluded from the exported-rule evaluation due to invalid or empty perception outputs. Could the authors clarify how these cases are handled in the full RIPA evaluation?

Thank you for the follow-up and for suggesting this controlled experiment.

Q1 — Missing conditions and final predictions

It is unclear whether MLLM verifier can correct these errors when these omissions are explicitly provided.

I think this raises concerns about the reliability of rule-grounded verification, so I suggest the authors to do follow-up controlled experiments by only the omitted size condition to the rule provided to the verifier while keeping other variables unchanged.

Explicitly providing the missing size condition can improve final predictions when the verifier checks it directly against the image. Simply appending it to the original prompt did not help. We tested both settings with Qwen3-VL-8B-Instruct.

Additional controlled experiments. We first checked the original 300-image counterfactual experiment. Fusion changed none of TDRL's predictions. The reported 3/26 size rejections and 15/26 color rejections therefore came from TDRL.

The original prompt also supplied erroneous Stage-1 size values. None of the 50 original positive images had extracted size values satisfying the required progression. After we appended the missing size condition, the verifier rejected all 50 size-violating variants, but also all 50 original positives. Final size rejections remained 3/26. Thus, supplying the condition alone did not resolve the errors under that prompt.

We then compared two arms that both checked the image without receiving Stage-1 size or color values. Both retained the same images, Stage-1 number/type atoms, TDRL predictions, and fusion weights. Both used the size-added arm's validation-selected weights, (1, 0, 4, 0). The arms differed only in whether the verifier received the size condition. Color remained omitted.

Final predictionsOriginal ruleOriginal rule + size
Size violations rejected: same 26 pairs1/2615/26
Size violations rejected: all 50 pairs15/5039/50
Accuracy on all 300 images208/300 (69.33%)221/300 (73.67%)

Adding size yielded 13 more correct predictions overall. The size-only arm accepted 22/26 original positives and 23/50 across all originals. When each arm used its own validation-selected weights, accuracy was 212/300 (70.67%) without size and 221/300 (73.67%) with size. The gain also held with separate validation tuning.

The verifier itself detected 0/26 size violations without the condition and 14/26 with it. With both size and color supplied and weights selected on validation data, final accuracy reached 229/300 (76.33%).

In a separate validation-based rule-refinement experiment, the exported rules recovered both conditions. They rejected 756/756 symbolic size/color violations and accepted 283/283 valid position-only variants. These are executable-rule results; the table above reports final predictions on edited images.

These controls show that the verifier can use an explicitly supplied condition to correct some final predictions. They also identify an important limitation: incorrect textual atoms can mislead verification. Direct visual checking improves correction, while some false negatives remain.

Q2 — Handling the 47 perception failures

Additionally, I am not quite sure about the 47 RAVEN 3x3 examples excluded from the exported-rule evaluation due to invalid or empty perception outputs. Could the authors clarify how these cases are handled in the full RIPA evaluation?

The 47 cases were also excluded from the original 8B cached-pipeline evaluation of full RIPA. Our previous response did not make this clear. We have now evaluated all 400 rows and report both failure-inclusive scores and results with an added visual fallback.

Evaluation audit and additional experiment. The test set contains 200 positive and 200 negative rows. Perception failed on 22 positives and 25 negatives. Preprocessing retained only successfully parsed outputs, leaving 353 rows: 178 positives and 175 negatives.

On these 353 rows, the original TDRL and RIPA scores were 237/353 (67.14%) and 244/353 (69.12%). Counting all 47 perception failures as errors gives 237/400 (59.25%) and 244/400 (61.00%).

We then added a fallback for the 47 failed rows. The same 8B model inspected three equally enlarged panels without Stage-1 atoms. It classified 24/47 correctly. Combining these predictions with the original RIPA outputs gives 268/400 (67.00%), with no rows excluded.

The additional validation-refined model classified 266/353 retained rows correctly. With the same fallback, its full-test result is 290/400 (72.50%).

The earlier 353-row scores were conditional on successful perception. The new 400-row results include all cases. The fallback restores coverage, although dense-object counting remains a source of error.

Reviewer_MmxPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  • RIPA combines probabilistic visual perception, explicit rule induction, and image-specific rule assessment.
    • An MLLM extracts probabilistic atoms describing objects, attributes, relations, and scene properties. A Template-Driven Differentiable Rule Learner (TDRL) learns class-specific rules from these atoms and exports selected rules to a fixed rule bank.
    • At inference, the MLLM receives the image, atoms, and retrieved rules. Its class scores and rule-support judgments are combined with TDRL predictions through agreement and conflict terms. The feedback changes the final decision but does not revise the learned rules.
  • The paper evaluates the framework on CLEVR-Hans, Kandinsky Patterns, and RAVEN-Custom.
    • The experiments include multiple MLLM backbones, comparisons with prompting and neuro-symbolic methods, atom-quality analysis, and ablations against hard atoms and TDRL-only prediction.
    • The full system reports improvements of 5.00–10.71 percentage points over TDRL-only prediction. These results provide evidence for the combined contribution of the subsequent MLLM and fusion stages.
Critical strengths
  1. The framework provides a useful way to connect explicit rules with visual evidence.

    • Preserving uncertain atom values rather than immediately converting them into hard facts is a reasonable design choice. The exported rules and rule-level support scores also provide intermediate representations that can be inspected.
    • The paper clearly distinguishes decision-level feedback from iterative rule revision. This helps explain what the proposed verification stage does and what it does not establish.
  2. The evaluation is not restricted to a single MLLM backbone.

    • The reported improvements extend across Qwen3-VL-8B, Qwen3-VL-235B, and GPT-5.4. This is a meaningful strength, although changing the backbone changes both perception and verification, rather than testing the verifier in isolation.
    • The TDRL-only comparison provides initial evidence that the feedback stage is useful. RAVEN-Custom improves from 65.96% to 76.67%, with the direct MLLM class-score weight set to zero. This supports the value of the rule-support pathway, though it does not establish superiority over alternative rescoring methods
Critical weaknesses
  1. The implementation may supply substantial parts of the rules that the method is meant to induce.

    • In chirp/models/tdrl.py, init_kandinsky_skeleton() initializes task-specific conditions. For red-triangle, these include red, triangle, existence, proximity, and shape/color differences. The supplied training script enables this initialization by default. This goes beyond specifying a generic rule vocabulary.
    • In chirp/reasoning/prompt_builder.py, _kandinsky_task_guidance() directly describes each positive concept. This text is selected by task name, not derived from the learned rule bank. The verifier may therefore apply a supplied target rule rather than assess a genuinely induced one.
    • Similar concerns extend beyond Kandinsky. Hans7 has class-specific skeletons and count thresholds. Hans3 has a conditional path that uses ground-truth rules for regularization and checkpoint selection. That path requires additional files. Its presence does not establish that it was used. The exact settings behind the reported results remain unclear.
  2. The evaluation does not sufficiently isolate rule induction, verification, or rule fidelity.

    • Unequal supervision. Appendix C gives Few-shot, CoT, and ILP-CoT 20 support images per class. TDRL trains on the training split. For Kandinsky, this means 40 support images versus 2,000 training images. Access to target-rule information is also unclear. The gains cannot be attributed to the method alone.
    • Unclear baseline provenance. Table 1 lists ILP-CoT Hans3 scores of 88.37/81.85 under Qwen3-VL-8B-Instruct. These exactly match the original ILP-CoT paper’s Qwen-7B results. That paper describes 300 images per class, divided into 60 groups. RIPA instead states a 20-image support protocol. Matching numbers do not prove reuse, but the discrepancy needs an explanation.
    • Incomplete component isolation. Table 3 supports the combined benefit of MLLM reasoning and fusion. It does not establish an advantage over ordinary MLLM rechecking under the same inference budget. The same verifier is also not tested with another rule learner.
    • Unverified rule fidelity. Figure 6 contains extra attributes and changed relations. Figure 8 omits size and color conditions from some RAVEN rules. Classification accuracy does not establish recovery of the target concept. Export also discretizes soft selectors and gates. The paper does not quantify whether the exported rules reproduce the soft TDRL predictions.
Questions for authors
  1. Highest priority: which target-specific components were used in the reported results?

    • Please specify whether task-specific initialization, hardcoded target-concept prompts, and ground-truth rule guidance were enabled for each dataset. Provide the configurations and prompts used for the reported runs, rather than only the current code defaults.
    • If these components were disabled, please provide the corresponding configuration or execution records. If they were enabled, please report a focused comparison without target-specific information while retaining the generic predicate vocabulary and template space.
    • Please also clarify whether the baselines received equivalent information. A baseline supplied with the same target rule would help assess known-rule application. However, such a comparison would not by itself establish rule induction from examples.
  2. Are the baseline results consistent with the stated models and supervision budgets?

    • Please explain the provenance of the ILP-CoT scores of 88.37/81.85. Were they rerun with Qwen3-VL-8B-Instruct, obtained from an existing checkpoint, or taken from prior work? Please identify the exact model and support protocol and distinguish these result sources throughout the tables.
    • To isolate the reasoning contribution, please provide one focused comparison with matched labeled-data access and target-rule information. A lower-data TDRL comparison or a shared-atom baseline would be informative. This does not require placing the entire training split into an MLLM prompt.
  3. After these protocol issues are clarified, what is the independent contribution of verification and rule export?

    • With TDRL and the atom cache fixed, please compare direct MLLM rescoring with rule-support verification under comparable image access and inference budgets. Please also clarify whether the soft-versus-hard atom comparison keeps the downstream pipeline fixed and specify the hardening procedure.
    • A further useful control would attach the same verification module to either NEUMANN or ILP-CoT. One learner on one benchmark would be sufficient for this diagnostic. This is secondary to resolving the target-information and baseline-protocol concerns.
    • Please distinguish target-rule recovery from export fidelity. Prediction agreement between the exported rules and soft TDRL would test fidelity. Counterfactual cases that alter irrelevant attributes or violate omitted conditions would provide additional evidence about rule recovery.

My main concern is the validity of the rule-induction comparison, not merely the absence of additional ablations. I would reconsider my recommendation if the authors resolve the protocol discrepancies and demonstrate that the main gains hold without supplied target concepts under comparable supervision.

6 comments

We sincerely thank you for your careful review of our work and for your questions.

Q1 — also addresses Weakness 1

Highest priority: which target-specific components were used in the reported results?

  • Please specify whether task-specific initialization, hardcoded target-concept prompts, and ground-truth rule guidance were enabled for each dataset. Provide the configurations and prompts used for the reported runs, rather than only the current code defaults.
  • If these components were disabled, please provide the corresponding configuration or execution records. If they were enabled, please report a focused comparison without target-specific information while retaining the generic predicate vocabulary and template space.
  • Please also clarify whether the baselines received equivalent information. A baseline supplied with the same target rule would help assess known-rule application. However, such a comparison would not by itself establish rule induction from examples.

The reported runs did use target-specific information. Hans3 used ground-truth-guided initialization and rule checking; Hans7 used a class-specific skeleton with count thresholds 2, 5, and 3. Kandinsky used task-specific TDRL initialization and written positive-class definitions in verifier prompts. RAVEN-Custom used fixed primitives without a preset target conjunction.

Additional guidance-removal experiments. We removed the specified sources of guidance in separate controls. Without rule guidance, Hans3 TDRL achieved 87.69% test accuracy. Removing the Hans7 skeleton and thresholds gave 87.52% mean accuracy across three seeds. Removing the written Kandinsky target definition from the verifier gave 87.81% mean accuracy across four tasks and 1,600 test images. With the RAVEN primitive library fixed, TDRL recovered both conditions of three new symbolic conjunctions in all nine runs.

The red-triangle prompt explicitly states that a red triangle must be near an object that is neither red nor triangular. This is target-specific information, and the Kandinsky prompt-removal control tests its contribution.

Additional target-rule baseline experiments. We also supplied target-rule information to baselines. Qwen3-VL-8B-Instruct received the target definition, image, and atom table, but no training examples; it classified 931/1,600 images correctly. ILP-CoT received the same 200 labeled support images per task as RIPA, with target clauses added to its candidates. With task predicates and group outputs fixed, adding the clauses changed neither its selected rules nor any of its 1,600 predictions; it remained at 986 correct. Together, these controls show what information was supplied, what performance remained after removing specific guidance, and whether explicitly giving target clauses changed the baselines.

Q2.1 — also addresses the baseline-provenance concern in Weakness 2

Please explain the provenance of the ILP-CoT scores of 88.37/81.85. Were they rerun with Qwen3-VL-8B-Instruct, obtained from an existing checkpoint, or taken from prior work? Please identify the exact model and support protocol and distinguish these result sources throughout the tables.

The 88.37% validation and 81.85% test scores came from Table 1 of the original ILP-CoT paper and correspond to Qwen-7B, not Qwen3-VL-8B-Instruct. The revised Table 1 reports our Qwen3-VL-8B-Instruct rerun: 89.33% validation and 79.87% test accuracy. We followed the original grouping protocol: 300 positive images per class, divided into 60 groups of five, with five negatives sampled for each group. All other ILP-CoT results in Tables 1 and 2 are from our experiments.

Q2.2 — also addresses the supervision concern in Weakness 2

To isolate the reasoning contribution, please provide one focused comparison with matched labeled-data access and target-rule information. A lower-data TDRL comparison or a shared-atom baseline would be informative. This does not require placing the entire training split into an MLLM prompt.

RIPA retains its advantage when labeled-data access is matched. Additional matched-budget experiment. We compared RIPA and ILP-CoT using identical nested, class-balanced support subsets, three seeds, and the same 400 test images per Kandinsky task. TDRL was retrained from its task-specific initialization; ILP-CoT retained its task-specific predicates and seed clauses. Results are means across four tasks and three seeds (Appendix F, Table 6).

Labeled images per task401002001,0002,000
RIPA (%)90.5890.5291.1091.0090.92
ILP-CoT (%)60.6060.7959.9661.6361.63

RIPA led in all 60 task–budget–seed comparisons. At 40 images per task, its mean advantage was 29.98 percentage points. The advantage therefore remains even when both methods use the same labeled examples.

Q3.1 — also addresses component isolation in Weakness 2

With TDRL and the atom cache fixed, please compare direct MLLM rescoring with rule-support verification under comparable image access and inference budgets. Please also clarify whether the soft-versus-hard atom comparison keeps the downstream pipeline fixed and specify the hardening procedure.

Verification can identify rule violations, but its isolated accuracy gain over direct rescoring is small in this RAVEN setting. Additional direct-rescoring control. The paper’s Table 3 compares the full reasoning-and-fusion stage with TDRL alone; it measures their combined contribution. We added a controlled RAVEN 2×2 comparison with TDRL, atoms, rules, and images fixed. Each arm used one Qwen3-VL-8B-Instruct call per row. Direct class rescoring matched TDRL at 299/400. A conflict penalty selected on separate validation data corrected three errors and introduced one, giving 301/400. Shuffling support scores across images reached at least 301/400 in 603/1,000 permutations, so this two-example gain alone does not establish an image-specific verification benefit. In paired image edits, however, the verifier’s written evidence identified 49/50 count violations and 48/50 type violations.

Additional soft-versus-hard atom experiment. To isolate the effect of atom uncertainty, we compared soft and hard inputs on red-triangle. We thresholded each cached shape, color, global-predicate, and object-relation probability independently at 0.5 (values ≥0.5 became 1; others became 0), while preserving object masks and class labels. With the same TDRL checkpoint, rule bank, and inference settings, changing only test inputs from soft to hard atoms reduced accuracy from 381/400 to 200/400. Retraining on hardened training atoms and testing on hardened test atoms yielded 258/400.

Q3.2

A further useful control would attach the same verification module to either NEUMANN or ILP-CoT. One learner on one benchmark would be sufficient for this diagnostic. This is secondary to resolving the target-information and baseline-protocol concerns.

The verification procedure also transfers to another rule learner. Additional verifier-transfer experiment. We attached the same Qwen3-VL-8B-Instruct verifier to ILP-CoT’s learned rule on red-triangle. When fusion proposed changing a positive prediction to negative, we accepted the change only if the rule also failed on cached object attributes and proximity relations. Accuracy increased from 215/400 (53.75%) to 322/400 (80.50%). This corrected 107 false positives while retaining all 155 previously correct positives.

Q3.3

Please distinguish target-rule recovery from export fidelity. Prediction agreement between the exported rules and soft TDRL would test fidelity. Counterfactual cases that alter irrelevant attributes or violate omitted conditions would provide additional evidence about rule recovery.

Exported rules reproduce most TDRL decisions, but they do not fully recover the target concept. Additional executable-rule and counterfactual experiments. We executed exported rules using the most probable attribute values and compared them with labels, soft TDRL, and final RIPA predictions. The RAVEN 3×3 test split had 400 rows; 47 produced invalid or empty structured outputs during perception. We evaluated all remaining 353 rows (178 positive, 175 negative).

RAVEN layoutRule accuracyAgreement with TDRLAgreement with final RIPA
2×275.00% (300/400)90.75% (363/400)91.00% (364/400)
3×368.27% (241/353)98.30% (347/353)97.45% (344/353)

To test omitted size and color conditions, we edited 378 valid RAVEN rows to create 756 invalid variants. The exports accepted all 756; TDRL accepted 709 and rejected 47. Both methods accepted all 283 valid position-only controls. On Hans3, changing the cylinder color or cube material—irrelevant to the target class—made the export reject all 40 scenes in each condition, while TDRL remained correct on 39/40 and 36/40, respectively. Thus, agreement measures how faithfully the export reproduces the classifier, while counterfactual edits reveal whether the exported rule captures the target conditions. The results show high decision fidelity but incomplete target-rule recovery.

Reviewer_MmxPOpenReview ↗

Response to the Rebuttal

Thank you for the detailed rebuttal. The clarification regarding the provenance of the baseline scores and the experiment attaching the verifier to the ILP-CoT red-triangle rule have successfully resolved my concerns on those specific points.

However, because the backbone and exact reference baselines for the ablation (removal) experiments were not clearly specified, it remains difficult to quantify the actual contribution of the target-specific guidance. Specifically, which "guidance-on" result should the 87.81% accuracy for Kandinsky be directly compared against?

Please provide the full-pipeline results under identical conditions with both the task-specific initialization and the hardcoded prompt definitions completely removed. The current prompt-only removal results do not sufficiently establish the framework's rule-induction performance in the absence of ground-truth target information, so I would appreciate further clarification on this matter.

Reviewer_7hdLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RIPA is a closed-loop framework that pairs an MLLM with neuro-symbolic rule learning for visual inductive reasoning. The MLLM first turns each image into probabilistic scene descriptions, which a differentiable rule learner (TDRL) converts into explicit, human-readable rules; at inference, the MLLM then checks whether each retrieved rule is actually supported by the image, and a fusion module combines both models' signals for the final prediction. The experiments are run on three visual reasoning benchmarks, with ablation studies examining the contribution of each component, and predictions can be traced back to the underlying rules and visual evidence.

Critical strengths
  1. The method design has several advantages. Probabilistic atoms retain perceptual uncertainty rather than forcing hard labels, and the hard-atom ablation shows that discretization lowers accuracy. Rules are exported and linked to visual evidence, making predictions more traceable than free-form explanations.

  2. The experiments cover three MLLM backbones, multiple tasks, weight sensitivity analyses, and component ablations.

Critical weaknesses
  1. It is unclear whether the closed loop can correct errors. The MLLM can only re-rank the Top-K classes that TDRL provides, so it cannot recover the correct class when TDRL misses it, and the paper never reports how often the correct class appears in these candidates. The same MLLM also handles both perception and verification, but there is no evidence that verification adds new information. For instance, the paper gives no error-correction rates and no comparison with a separate verifier.

  2. The RAVEN-Custom benchmark and the TDRL templates use the same set of rules. The benchmark is built from transformation rules that closely match TDRL’s templates, so strong results may simply mean the model matches the templates rather than learning new rules, and the authors do not show that the two sets of rules actually differ.

Overall, this paper proposes an interesting inductive reasoning approach RIPA, which pairs MLLM predictions with an explicit symbolic rule bank, but the current evidence does not establish that the closed loop adds much beyond TDRL alone. The MLLM only re-ranks TDRL’s Top-K candidates and cannot recover a class that TDRL excludes. It is also unclear whether verification adds anything beyond the perception stage, and whether the RAVEN-Custom benchmark shares its rule vocabulary with the TDRL templates. As the paper stands, the interpretable rule structure is its clearest contribution, while the value of the second-stage reasoning remains an open question.

Questions for authors
  1. How often does TDRL’s Top-K set include the correct class, and what happens when it does not? Does verification add anything beyond the first perception step, for example, how often does it catch perception errors, and what happens with a separate verifier?

  2. Can you show that the benchmark rules and the TDRL templates are not the same, for instance, how does the model do on tasks whose rules the templates do not cover?

3 comments

We sincerely thank you for your careful review of our work and for your questions.

Q1 — also addresses Weakness 1

How often does TDRL’s Top-K set include the correct class, and what happens when it does not? Does verification add anything beyond the first perception step, for example, how often does it catch perception errors, and what happens with a separate verifier?

The verifier can correct some visual facts when it can inspect the image, but it cannot recover a class excluded from TDRL’s candidate set. Additional candidate-coverage experiment. On Hans7, however, the correct class appeared in TDRL’s top four for 5,226/5,250 images (99.54%). We then passed all seven classes to the same Qwen3-VL-8B-Instruct verifier. It recovered none of the remaining 24 cases. Top-4 and top-7 produced identical predictions: 4,333/5,250 correct (82.53%).

Additional perception-error correction experiment. To test correction beyond the supplied atoms, we selected 100 held-out RAVEN 2×2 rows whose object counts and shape types matched the scene annotations. We corrupted either the count or shape type in the last panel and compared verifier calls with identical atoms and rules, differing only in access to the original image. A correction counted only when the verifier returned the exact original value.

Corrupted attributeWithout imageWith image
Shape type0/10020/100
Object count2/1008/100

With unaltered atoms and image access, the verifier reported both attributes correctly on 42/100 rows. Image access did not change final classification accuracy under either corruption condition. Additional independent-verifier evaluation. We evaluated 934 Hans3 examples with Qwen3-VL-8B-Instruct as verifier. With priors, rules, and fusion weights fixed, RIPA correctly classified 847/934 examples. Thus, the verifier added image-based evidence at the attribute level, but this evidence did not translate into higher final classification accuracy in the RAVEN corruption test.

Q2 — also addresses Weakness 2

Can you show that the benchmark rules and the TDRL templates are not the same, for instance, how does the model do on tasks whose rules the templates do not cover?

Yes. The benchmark rules are not identical to the TDRL templates. The templates provide basic operations, while labeled examples determine which attributes and conditions define each class. We tested both new combinations of existing operations and a relation that the original template set cannot express.

Additional experiment: new conjunctions within the original template space. We kept the original templates fixed and created three new tasks, each requiring two conditions (e.g., size increases while color stays constant). Each negative violated exactly one condition of its paired positive. Across three tasks and three seeds, TDRL exported both required conditions in all nine runs. Executing the exports achieved 400/400 on each task; soft TDRL achieved 400/400, 400/400, and 333/400.

Additional experiment: a relation outside the original template space. We defined a positive row as one in which the third-panel object was larger than the first, regardless of the middle panel. Swapping the endpoints reversed the label but preserved all original template scores, limiting the original model to 200/400. We then added an operator that compares any two panels by count, size, or color. Using class labels alone, the model learned the correct attribute, panel pair, and direction. Both soft TDRL and the exported rule achieved 400/400 at all three seeds. With this operator fixed, three alternative labelings of the same symbolic rows each yielded the corresponding rule and 576/576 correct per seed.

Together, these results show that TDRL learns new combinations within the existing template space. A relation outside that space becomes learnable when the operator set is extended.

Thanks for the detailed responses. I appreciate the new results for Q2. This largely addresses my previous concern on template cloning. However, I think that Q1 is not sufficiently addressed, since it justifies that the current error correction mechanism is not sufficiently designed to handle all kinds of perceptual errors, which is crucial to visual induction. Due to this reason, I maintain my previous score.

Thank you for confirming that our new experiments largely address Q2. We have now focused on the remaining concern about perceptual error correction.

Q1 — Perceptual error correction and final predictions

However, I think that Q1 is not sufficiently addressed, since it justifies that the current error correction mechanism is not sufficiently designed to handle all kinds of perceptual errors, which is crucial to visual induction.

Our new experiments show that visual correction can improve final predictions on both RAVEN and Kandinsky. We added atom feedback to the RAVEN inference process and improved visual clause verification on Kandinsky. These experiments test controlled attribute corruptions and perception errors produced by the 8B model itself.

Additional experiment 1: RAVEN atom repair. We reused the same 100 held-out RAVEN 2×2 rows, whose original count and shape-type atoms matched the annotations. The Qwen3-VL-8B-Instruct verifier inspects an enlarged third panel without receiving the corrupted attribute values. We write the recovered values into the corresponding atom slots and rerun the same TDRL model. This adds attribute feedback to our earlier verification-and-fusion probe.

Final correct predictions: 100 rowsTDRL before repairVisual audit + atom repair
Clean inputs70/10070/100
Count corruption50/10070/100
Type corruption45/10070/100

The visual audit recovers 100/100 corrupted counts and 100/100 corrupted shape types. Final accuracy returns to the 70/100 clean-input baseline under both corruptions. This gives 20 and 25 more correct predictions than TDRL on the respective corrupted inputs. On clean rows, the procedure preserves both attributes in all 100 cases and introduces no classification regressions.

The recovered attributes therefore affect the learner's decisions. The combined visual audit and atom feedback recover the accuracy lost to both tested corruptions.

Additional experiment 2: Kandinsky perception errors. We evaluated 700 red-triangle counterfactual images under three paired seeds, giving 2,100 evaluations. The edits changed color, shape, distance, or position. The 8B model extracted atoms from these images normally; we did not inject errors into the atoms.

We identified 414 evaluations where TDRL was correct with scene-annotation atoms but wrong with cached 8B atoms. This isolates a subset of errors attributable to perception.

We removed numeric prior values from the verifier prompt. We gated clause verification using support-set precision ≥ 0.90 and positive coverage ≥ 0.50. With the conflict weight set to δ = 0.8, visual verification corrected 164/414 perception-related errors (39.6%). Across all 2,100 evaluations, accuracy increased from 74.00% to 82.19%, with improvements in all three seeds.

This experiment shows that visual verification corrects final decisions affected by errors in the model's own perception. The improvement extends beyond the manually corrupted atoms used in the RAVEN control.

Together, these results show that visual verification can improve final predictions through atom feedback and clause-level checks. The evidence covers both controlled attribute corruptions and errors produced by the perception model itself.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.