acceptodds
Under review as a conference paper at ICLR 2027

RuleRouter: Sparse, Traceable Natural-Language Interventions for Frozen Vision-Language Models

Abstract

Natural-language rules can target structured errors in frozen vision-language models, yet unconditional application often overturns correct predictions. We introduce RuleRouter, a sparse and traceable intervention layer that derives contrastive rules from development-only confusion evidence and applies a proposed correction only when at least four of five benefit–harm router replicas agree that predicted benefit exceeds harm. The rules, routers, and operating points are frozen before their corresponding confirmation evaluations. Relative to a supervised logit-adapted System-1, RuleRouter improves Macro-F1 on EuroSAT-RGB by 5.53 percentage points (95% paired-bootstrap CI [3.90, 7.06]) at 16.8% intervention coverage. On an independent AID-30 final-confirm set, a separately frozen pipeline improves its prespecified reporting comparator by 3.17 points (CI [2.26, 4.15]), while changing 7.83% of its predictions. So2Sat serves as a low-base-performance stress test and shows a smaller 0.46-point gain. Across these evaluations, selective routing produces more corrections than harms and avoids the degradation caused by unconditional rule fusion. Equal-label visual probes remain more accurate overall, while matched semantic controls favor confusion-driven rules on EuroSAT but provide no corresponding evidence on So2Sat. RuleRouter therefore addresses settings that require sparse changes to a frozen predictor, explicit coverage and harm accounting, and an inspectable textual record for each intervention.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, natural-language rules, selective intervention, utility-aware routing, interpretable machine learning, remote sensing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_oqVsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

this paper presents RuleRouter, which selectively replaces a frozen VLM's predictions using confusion-derived textual rules and an ensemble of benefit–harm routers. the authors use an extra development set to decide the rules, adaptation, and intervention thresholds before evaluation. Reported Macro-F1 gains are 5.53 percentage points on EuroSAT, 3.17 on AID, and 0.46 on the low-performing So2Sat setting

Critical strengths

I think the biggest strength of the paper is just the reported scores with very comprehensive metrics, and multiple runs to mitigate the noise.

Critical weaknesses
  1. the biggest problem of the paper is presentation. Vague definitions like v1, v2 everywhere, the figure 2 as the main pipeline is hard to read with misaligned formats and complex annotations. In general, I wouldn't put the presentation of this paper to a 'ready-for-review' tier.
  2. The different routing components lack separate benefit. Shared-target comparisons do not establish gains from dual heads or replica averaging, and voting requirements from 1/5 through 4/5 produce identical predictions at the frozen cutoff, I'm not convinced each component actually works as reported together
Questions for authors
  1. At identical realized intervention coverage, does the full router reduce harm wrt benefit-only routing, with paired uncertainty?
  2. Can you quantify how often activated clauses are visually correct and discriminative for their class pair?
Reviewer_3pWUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper aims to answer this question - how do you fix a frozen model’s recurring classification mistakes without retraining it, without breaking predictions it already gets right, and while leaving a readable record of every change?

During an offline analysis, each class gets a confused class assigned based on the likeliest misclassification of that class. Hints are written for those cases in the form of clarifying questions that help avoid the mistake identified. On a second labeled dataset, it is checked whether switching to the hinted class would have fixed the error or broken correct answers. A small model is trained to predict that.

At inference time, per class classification is done and highest scoring class (after nudging the probabilities for correction) is picked. From the several hints, the highest scoring hint picks the alternate class. If the verdict is flipped, the hint is stored as the reason.

The paper shows macro-F1 gains across 3 different datasets while changing about 8-17% of answers. Applying hints to every image without a checker makes results worse.

Critical strengths

The idea is interesting given the attention back to classifiers in the form of system 1 type of models. Using hints and checker to sanity check zero shot performance and improve them is relevant to the current trend.

The paper's ablations are good. The authors compare against baselines trained on the same labels, a visual router held to the same number of switches as RuleRouter, shuffled and random hints, applying the hints without checker, etc.

Critical weaknesses

The paper has several weaknesses:

  • The approach doesn't beat simpler alternatives trained on the same labels as shown by their own evaluation: A checker with no hints at all beats RuleRouter on EuroSAT. Selective defer scores higher on EuroSAT and AID. Even a plain linear probe on same model's image features high scores on both EuroSAT and AID. The gains from hints therefore are really weak (also evidenced by the fact that shuffled or random hints still improve EuroSAT).

  • The downstream use of the hints are also not of significance. Some hints are nonsensical, for example, "the letter a is present". The paper itself shows 48% of the hints are potentially irrelevant.

While the use of clarifying questions to model might be relevant, the overall solution might have to be revisited. Prompt optimization techniques might already show gains over this approach, so how to distill that into hints might be helpful.

Questions for authors

How do the hints contribute? How does the checker perform without the class-identity inputs, using only hint scores, confidence, and margin? Is AID’s “reporting S1” the raw or the adjusted classifier, and why does AID use a different classification prompt and a separate fallback?

Why are hints triggered by the raw answer when the switch replaces the adjusted answer? How often do the two differ on the test sets?

Does RuleRouter still add anything on top of a better System-1, for example an optimized prompt, a single multiple-choice prompt, or higher-resolution inputs?

Usefulness of the traces. Is there any evidence, even a small human study, that the saved hints help someone spot wrong switches?

Reviewer_C1MtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RuleRouter, a selective intervention layer for a frozen Qwen2.5-VL-7B scene classifier. Specifically, the VLM writes contrastive clauses for each raw-S1 confusion edge, a temperature-and-class-bias adapter defines adapted S1, and five bootstrap benefit/harm routers decide (4-of-5 votes plus a development-selected cutoff) whether to replace adapted S1 with the rule's proposal. Paired Macro-F1 gains are 5.53, 3.17, and 0.46 pp on EuroSAT, AID-30, and So2Sat at 6.5 to 16.8% coverage.

Critical strengths
  1. The protocol is careful and unusually honest. Splits are disjoint, cutoffs are frozen before testing, and the paper keeps evidence that cuts against it, such as the stronger visual probes (Table 4) and the unedited rule catalog (Tables 26 to 27). Every claim can be checked, which counts for a lot in my assessment.
  2. Routing on predicted benefit versus harm against a named fallback (Eqs. 5 to 6) is a framing others can reuse. The evidence for selectivity itself is solid: the full policy beats unconditional fusion by 11.38 pp on EuroSAT and 4.50 pp on So2Sat (Tables 12 to 13).
Critical weaknesses
  1. The most important opportunity, I think, is to show that the rule text carries signal, since traceability is the paper's main selling point. Text-free routers match or beat RuleRouter (matched-gate visual router 0.3777 vs. 0.3721 on EuroSAT; no-text top-2 router 0.0668 vs. 0.0643 on So2Sat; Table 4). Eq. 7 gives the router one-hot adapted and proposed classes, so it can learn a class-pair switching table whatever the clauses say. Many clauses also ignore scene content: 47 of 50 So2Sat rules fire on raw S1 = Sparsely built, including "the image contains a keyboard" (Table 27), and a single "the image is dark" clause yields both a correction and a wrong-to-wrong change (Fig. 11c,d). Differential descriptions for confusable classes [1] and language-based rectification [2] are close prior work and natural baselines. A constant-clause ablation, plus a small blinded study of whether the traces help users catch harmful interventions, would settle this.

  2. The comparators are weak, and the strongest one is missing for AID. Adapted S1 reaches only 32.90% accuracy on EuroSAT (Table 7), while a probe on the same cached S1 evidence reaches 0.6122 Macro-F1 (Table 14). So2Sat inputs are nearly black, and Macro-F1 stays near 0.06 (App. J). Table 18 has no adapted-S1 row for AID, yet Sec. 4.3 reports that adapted S1 beat V1 by 6.77 pp on historical AID. Adding that row, and a stronger S1 such as multiple-choice prompting or RemoteCLIP (cited at line 80), would show whether the gains hold.

[1] Esfandiarpoor, Reza, and Stephen H. Bach. "Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification." ICLR, 2024.
[2] Zhang, Yuhui, et al. "Diagnosing and Rectifying Vision Models using Language." ICLR, 2023.

Questions for authors
  1. Is the AID final-confirm pipeline the V1 or the V2 router? Tables 1 and 2 both show n = 2,400, a 6.50% gate, and 104 corrections; please confirm the two AID sets are disjoint.
  2. At the frozen EuroSAT cutoff, what changes if s_r and e_r are set to constants?

I'm willing to raise my score if adapted S1 on AID does not erase the gain and the clause ablation shows the text matters.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.