acceptodds
Under review as a conference paper at ICLR 2027

When Is One Importance Ranking Enough? Auditing Ranking Reuse Across Decisions

Abstract

Reusing one importance ranking across lesion-based functional diagnosis, concept correction, vulnerability screening, and specialization audits compresses distinct decision utilities into a common ordering. We evaluate this decision compression by separating compatibility, consequence, auditability, and authorization. Conflict structure does not determine sharing cost, and reliable measurement alone does not authorize reuse. Controlled redundancy separates single-unit effects from group responsibility; a construction shows that identical complete strict preferences can yield different minimum sharing costs even with fixed per-decision utility ranges. On CUB, same-feature common-linear and decision-specific linear rankers incur held-out mean normalized regrets of 0.4346 and 0.0785. A penalty persists under development-only stronger search and exhaustive optimization in an eight-concept test-utility control. A five-dataset non-CBM replication supports positive raw cost but reveals a denominator-sensitive normalized reversal. A prespecified blind AwA2 audit and separately frozen exploratory CelebA replication attain family-pair ordering accuracies of 0.956 and 0.929. Using 40% of audit observations, median accuracy retains over 90% of full-audit accuracy, but only 60.0–67.7% of subsampling runs meet the declared 0.85 criterion. In a 1,000-family LAD study, frozen direct-audit and cheap-proxy policies each select 100 of 500 held-out families at a fixed 20% reuse quota. At tolerance , both record 0/100 observed violations and overall normalized regret near 0.039, versus 0.3625 under always sharing. This improvement combines reduced reuse and selection; direct-audit superiority over the cheap proxy is not established. Separate prospective studies yield both certification and abstention under risk and coverage requirements. Ranking reuse therefore requires consequence evaluation and policy-specific validation, not confidence in an importance score alone.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, concept bottleneck models, importance ranking, ranking reuse, functional auditing, decision-dependent evaluation, interventions, selective reuse

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_uXNfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines if a single importance ranking can be reused across multiple downstream decisions. The paper introduces a framework that distinguishes compatibility, consequence, auditability, and authorization. They present experiments comparing shared and decision-specific rankings and measure sharing costs and selective-reuse policies. The central finding is that decision-specific rankings can outperform shared rankings and that ranking reuse should be evaluated through its downstream consequences.

Critical strengths
  • The paper considers a relevant practical question concerning the reuse of importance rankings across different decisions tasks.
  • The authors conduct experiments using multiple datasets and provide detailed descriptions of their experimental setup.
Critical weaknesses
  • The paper lacks significant technical novelty. The two theoretical propositions are simple observations. I do not see any substantive new theoretical or algorithmic contributions. Much of the ideas in the paper are established ideas from the field of decision-focused learning.
  • The main experimental results are also unsurprising. They are the expected consequence of relaxing an optimization constraint. The experiments do not establish any interesting phenomenon or meaningful generalizable insight.
  • The paper is poorly written. The work introduces unusual terminology and would benefit from leveraging standardized language from the field. The experiments use disparate settings with varying objectives and normalization procedures, making it difficult to identify what the results are meant to show and why they are interesting.
  • Most of the paper is spent formalizing the framework via excessive notation and terminology. These largely correspond to existing concepts in preference aggregation, decision-focused learning, and statistical estimation, but this connection is omitted from the paper. It is unclear what novel methods the paper contributes.
Questions for authors
  1. Are there any new algorithmic or technical insights that can be drawn from the results?
  2. How do the terminology, models, and problem formulations in this paper relate to established formulations in the broader AI literature? What does the proposed framework contribute beyond these existing approaches?
Reviewer_eEN5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

As far as I can tell, the authors ask whether one importance ranking can be reused across several decisions. They propose a test for when a single shared ranking works and a measure of what is lost when it does not. I could not tell what is being ranked or what the ranking is for.

Critical strengths

None that I could identify.

Critical weaknesses

This paper reads as language-model output that no one checked. Calling it unclear understates the problem. I could not get far enough into it to understand what it is trying to do, let alone judge whether its contribution is sound or significant. I do not think it can be salvaged in its current form, and I do not think an average reader could work out what the contribution is. In particular:

  • W1. The paper never defines an importance ranking, which is its central object.
  • W2. It does not say which kind of ranking it studies, who uses the ranking, or for what decision.
  • W3. The abstract is too long, and the introduction is illegible.
  • W4. It does not explain how memory-style orderings and concept bottlenecks relate to the main contribution.
Questions for authors
  • Q1. Please give one concrete, real-world example of the problem you are solving.
  • Q2. In that example, what is being ranked, who uses the ranking, and what decision do they make with it?
  • Q3. In plain terms, what is the problem, and what is your solution?
  • Q4. Why should anyone use your method?
  • Q5. How does this paper advance science?
Reviewer_S71cOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

I found the paper very difficult to follow. The abstract does not clearly state the problem being addressed, the contribution, or the outcome of the study. Even after several readings, I could not make head or tail of much of the paper. To illustrate my concern, I analyze one part of the paper below.

Critical strengths

None

Critical weaknesses

Consider Section 3.1. The section starts by defining a protocol Π, which contains units, actions, budgets, utility scales, comparator classes, and sampling units. None of these objects is defined in the main text, either formally or by an example. Some definitions appear in Appendix C.1, and even the appendix gives no examples. This is followed by the sentence “Units belong to a model,” and it is not clear how units relate to a model. Next, we learn about a family F in the sentence “A family F is a cross-model evaluation unit with study-specific aggregation.” What is a family? What is a cross-model evaluation unit? What is study-specific aggregation? Next, the section defines a function U with many subscripts, which takes an input S. What does this set contain? What is an ordering? What do the subscripts b and d denote? After Equation 1, the weights are said to have certain properties, but the weights w_{d,b} are never introduced.

Then we get the sentence “Raw, menu-regret, and anchor-normalized analyses remain distinct.” What are the raw, menu-regret, and anchor-normalized analyses?

I could continue, but these examples show that the main text is not self-contained. Understanding even the basic framework requires constant reference to the appendix, which makes the paper very hard to read.

I strongly recommend careful rewriting of the paper.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.