- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RIPA combines MLLM perception with differentiable rule induction. It extracts probabilistic atoms, learns class-specific rules with TDRL, and asks an MLLM to verify retrieved rules. A fusion module combines class scores with rule agreement and conflict. Experiments cover CLEVR-Hans, Kandinsky Patterns, and a fixed-rule binary variant of RAVEN. The paper reports improved classification and exports readable rules.
Critical strengths
-
The integration is well motivated. Retaining uncertain atoms avoids early hard decisions. Explicit rule retrieval and verification provide inspectable intermediate evidence.
-
The evaluation includes three MLLM backbones, hard-atom and TDRL-only ablations, and oracle-atom analysis. The full system improves over TDRL alone across the reported datasets. The supplementary code and dataset generators also help reproducibility.
Critical weaknesses
-
Task-specific rule knowledge confounds the induction claim. The supplied Hans7 configuration enables init_hans7_skeleton(). This initializes class-specific rule structures and ground-truth count thresholds, including 2, 5, and 3 (chirp/models/tdrl.py). Kandinsky initialization also presets task-specific predicates and attributes. These parameters remain learnable, but the initialization supplies substantial prior knowledge. Separately, _kandinsky_task_guidance() inserts each task's positive-pattern definition into the verification prompt (chirp/reasoning/prompt_builder.py). For example, it specifies the red-triangle rule and its required nearby object. The paper does not clearly disclose these inputs or isolate their contribution. Please clarify whether the reported runs used these defaults. Without controls, the gains cannot be attributed cleanly to learned rules and verification.
-
The comparison with ILP-CoT uses unequal supervision. Appendix C gives ILP-CoT 20 support images per class. TDRL uses the full training split, including 2,000 images per Kandinsky task and RAVEN configuration. Thus, on these binary tasks, the comparison uses 40 versus 2,000 labeled training images. A matched-data comparison is needed to separate methodological gains from additional supervision.
Questions for authors
-
Which reported runs used the class-specific initialization and hand-written rule guidance? Please provide results without them and baselines given the same rule knowledge.
-
How does RIPA compare with ILP-CoT under identical labeled support budgets? A learning curve would help assess the contribution of extra data.
-
How faithfully do the exported rules reproduce the soft model's predictions? Figure 8 omits size and color conditions in the 2x2 and 3x3 exports. Please report executable-rule accuracy and agreement with TDRL and final predictions.
We sincerely thank you for your careful review of our work and for your questions.
Q1 and W1
Hans3 used ground-truth-guided initialization and ground-truth rule checking. Hans7 used a class-specific skeleton with count thresholds 2, 5, and 3. Kandinsky used task-specific TDRL initialization and written positive-class definitions in verifier prompts. For example, the red-triangle prompt specifies a red triangle near an object that is neither red nor triangular.
Additional guidance-removal experiments. Removing Hans3 rule guidance gave 87.69% test accuracy. On Hans7, three paired seeds gave 84.95% mean accuracy with the skeleton and 87.52% without it. All runs without the skeleton learned that class 3 contains small brown, green, and purple objects, but did not export the additional requirement of at least five small objects. Removing the written Kandinsky target definition gave 87.81% mean accuracy across four tasks and 1,600 test images.
Additional target-rule baseline experiments. We also supplied target-rule information to baselines. Qwen3-VL-8B-Instruct received the target definition, image, and atom table, but no training examples; it classified 931/1,600 correctly. ILP-CoT received the same 200 support images per task as RIPA, with target clauses added to its candidates. It classified 986/1,600 correctly. Removing those clauses while retaining task-specific predicates changed neither the selected rules nor any prediction. These controls show that performance remains after the tested guidance is removed, while supplying explicit target clauses did not change ILP-CoT’s result.
Q2 and W2
RIPA retains its advantage when both methods use identical labeled support sets. Additional matched-budget learning-curve experiment. We compared both methods using the same nested, class-balanced subsets at five budgets, three seeds, and the same 400 test images per Kandinsky task. TDRL was retrained from its task-specific initialization; ILP-CoT retained its task-specific predicates and seed clauses (Appendix F, Table 6, Figure 9).
RIPA achieved higher mean accuracy on all four tasks at every budget. Its advantage at 40 images per task was 29.98 points. Increasing the budget from 40 to 2,000 raised its mean accuracy by only 0.34 points. Thus, the gap remains when the labeled-data budget is matched.
Q3
The exported rules reproduce most model decisions, but this does not establish that they recover all target conditions. Additional executable-rule and counterfactual experiments. We executed the exported conjunctions using the most probable attribute values and compared them with labels, soft TDRL, and final RIPA predictions. The RAVEN 3×3 test split contained 400 rows; 47 had invalid or empty perception outputs. We evaluated the remaining 353 rows (178 positive, 175 negative).
On 3×3, the export reproduced 347/353 TDRL decisions, including errors. We then changed only the final panel’s size or color annotations in 378 valid rows, creating 756 invalid variants. The exports accepted all 756; TDRL accepted 709. Both methods accepted all 283 valid position-only controls. Thus, the exports are largely faithful to classifier behavior, including some errors, while the counterfactuals show that recovery of the full target concept remains incomplete.