acceptodds
Under review as a conference paper at ICLR 2027

Who Gets to Constrain the Decision? Authority-Conditioned Graph Fusion for Clinical Multimodal Learning

Abstract

Multimodal feature fusion combines heterogeneous evidence for structured decision-making, but clinically motivated scenarios raise a fundamental question: when predictive strength, reliability, and evidential authority conflict, which evidence source should constrain the final output? We formulate this problem as an authority-asymmetric multimodal learning process and propose the Authority-Consistent Fusion (ACF) framework that treats authority as a task-dependent specification. ACF includes three components: a label-free Authority Specification Compiler to derive explicit authority constraints, an Authority-Protected Graph Mixer to prevent unauthorized information propagation, and an Authority-Consistent Projection to enforce admissible outputs under missing, invalid, and conflicting evidence. Under explicit assumptions, we establish relation-preservation, fixed-parameter non-interference, and conditional squared-loss guarantees for the proposed framework. Experiments on the UTSW MRI-to-semantic retrieval benchmark demonstrate that ACF achieves comparable calibration to information-matched learned joint graph fusion, with a clean Brier score difference within 7.0x10-5. Controlled evaluations further show that ACF maintains a zero S1 authority-violation rate under activated relations and incurs Brier increases of only 0.084–0.189 under incorrect authority specifications. Additional matched experiments attribute 6.49 and 2.48 mAP improvements to the evaluated geometric and communication configurations, respectively. ACF establishes evidential authority as an explicit decision variable and provides a principled foundation for reliable and accountable multimodal decision-making when heterogeneous evidence sources disagree.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Multimodal Learning, Authority-Consistent Fusion, Graph Neural Networks, Constrained Learning, Trustworthy Machine Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_14nYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper studies the role of evidential authority in clinical multimodal learning. Based on their theoretical insights, the authors propose ACF, which combines authority specification, protected graph fusion, and constrained projection to control how different modalities influence the final decision. The paper provides theoretical analysis and evaluates the method on controlled tasks and an MRI-to-semantic retrieval benchmark.

Critical strengths

Strengths

  1. The paper discusses the distinction between predictive utility, reliability, and authority, which is a relevant question for clinical multimodal learning.
  2. The authors provide several controlled experiments to examine the behavior of different components under predefined authority constraints.
Critical weaknesses

Weaknesses

  1. But now multi-modal learing in medical domain often consider modalities over two, so a main concern is that - does this proposed framework generalize to high modality learning scenarios? It will be better to see more limitation analysis or reltaed work discussing such efforts in the community [1,2].
  2. My main concern is that the core contribution is not sufficiently validated on real-world data. The authority mechanism is mainly evaluated on synthetic tasks, while the UTSW experiments focus on representation geometry and graph communication. I wonder why the proposed authority mechanism is not directly evaluated on a real clinical prediction task?
  3. I also question whether the proposed protected graph is necessary. Can you elaborate more on the motivation of this design?
  4. Another concern is the assumption of predefined authority rules. I think determining which modality should have authority is itself a difficult problem in real clinical applications. It may vary across task, for example, for cariovascular diseases detection, ECG might be useful. And for pleural effusion, Chest Xray might be more userful. I wonder whether the proposed framework is useful without reliable authority specifications.

[1] Liang P P, Lyu Y, Fan X, et al. High-modality multimodal transformer: Quantifying modality & interaction heterogeneity for high-modality representation learning[J]. arXiv preprint arXiv:2203.01311, 2022.

[2] Huang Y, Du C, Xue Z, et al. What makes multi-modal learning better than single (provably)[J]. Advances in Neural Information Processing Systems, 2021, 34: 10944-10956.

[3] Wu C, Shuai Z, Shen L. Rethinking Medical High-Modality Learning Under Missingness--A Long-Tailed Distribution Perspective[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 5641-5651.

Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer 14nY- Part 1

Thank you for recognizing the distinction between evidential authority, predictive utility, and reliability, and for raising questions about modality scale and clinical applicability. We clarify the supported scope, the role of protected computation, and the conditions for using authority specifications below.

All page and line references below refer to the revised manuscript PDF.

1. Extension to more modalities

Both AIS tasks use four sources, and the formulation admits MM sources with relation-specific authorized sets. S1 specifies one relation per record, and S2 specifies two, with designated sources varying across records. Each relation activates when its supporting evidence is valid and available. The non-interference argument follows the permitted computational dependencies for each relation at fixed parameters, compilation, and authorized inputs.

The empirical scope is four sources and up to two simultaneous relations. At larger source counts, dense attention increases the pairwise computation per relation; predictive performance at that scale would require further evaluation.

The suggested works address complementary aspects of this extension: HighMMT studies modality and interaction heterogeneity [1], Huang et al. analyze conditions for predictive benefit from additional modalities [2], and REMIND examines long-tailed modality combinations under medical missingness [3]. ACF addresses the supplied task specification governing computational dependencies and output relations. The revised Section 3.1 states the evaluated scope (p. 5, lines 235–237), and Section 3.2 gives the general MM-source formulation (p. 5, lines 241–255).

2. Evidence from real clinical data

The evaluation separates controlled tests of authority from real-data tests of the fusion backbone. AIS provides known decision relations and controlled changes to source assignments, validity, reliability, and disagreement. This permits testing whether the mechanism follows a supplied relation while measuring the predictive consequences of that relation. The reason for using AIS is this experimental control over authority and its alternatives.

UTSW evaluates MRI-to-semantic retrieval and attributes differences to representation and communication choices. Its matched comparisons yield 6.485 mAP points for the evaluated SPD-versus-Euclidean configuration and 2.482 points for uniform-versus-identity communication (Appendix Table B.12). A direct clinical authority evaluation would additionally require a validated mapping from evidence statements to decision constraints and testing under real-source disagreement. The present authority-specific conclusions concern the controlled AIS setting.

The revised Section 3.1 distinguishes these settings (p. 5, lines 235–237), the abstract assigns the AIS and UTSW results accordingly (p. 1, lines 024–029), and the conclusion states the clinical validation requirement (p. 8, lines 421–423).

References

[1] Liang et al. *High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning.

[2] Huang et al. What Makes Multi-Modal Learning Better than Single (Provably). NeurIPS 2021.

[3] Wu et al. *Rethinking Medical High-Modality Learning Under Missingness—A Long-Tailed Distribution Perspective.

AuthorsOpenReview ↗

Response to Reviewer 14nY- Part 2

3. Motivation for the protected graph

APGM provides a stable reference state for auditing declared source dependencies while the final prediction uses auxiliary context. In an implementation audit, one can hold the authorized evidence, parameters, and compiled specification fixed, vary the auxiliary inputs, and inspect the designated state before readout. A change in that state reveals an auxiliary dependency across the declared boundary, even when the final prediction satisfies every output constraint. APGM enforces this dependency requirement; ACP enforces the output relations. Tasks requiring output compliance alone can use projected joint fusion.

Appendix Table B.8 provides a concrete check: a deliberately inserted auxiliary bypass produces mean state shifts of 0.062627 on S1 and 0.056688 on S2 while output AVR remains 0. Thus, an output-compliance check can pass while the designated state's source dependencies have changed.

The saved S1 reliability interventions also show retained auxiliary predictive utility: ACF residual Brier decreases from 0.281343 to 0.076133 as auxiliary reliability increases from 0.60 to 0.95, at fixed η=0\eta=0 and κ=0\kappa=0, while its designated-state shift remains 0. In the strongest tested S1 conflict, mean residual Brier is 0.308853 for ACF and 0.288601 for learned-joint, with zero AVR for both. The evidence supports the audit property together with useful auxiliary prediction in the reliability settings; predictive performance depends on the intervention.

The revised Overview states the task requirement (pp. 4–5, lines 214–219), and the Illustrative example shows how auxiliary information reaches the final prediction while the designated state remains protected (p. 5, lines 221–234). Table 3 explicitly labels the deliberate-bypass audit (p. 8, lines 388–389). The reliability and conflict comparisons above are presented in this response.

4. Dependence on reliable authority specifications

With usable inputs and no active authority relation, ACF returns its unconstrained prediction; its authority-specific guarantees concern the relations supplied by the task protocol. The reviewer's examples highlight why the target and decision relation matter. The protocol determines which evidence statement supports a particular exclusion or probability order; validity and availability determine whether that relation is activated.

An incorrect policy that remains active has a different effect from an inactive relation. Table 1 measures its predictive cost even when the model satisfies the supplied constraints.

Table 1: Predictive sensitivity to authority-policy misspecification (Appendix Table B.5). Entries are mean Brier scores over five data seeds, each averaging three model initializations. Both policies yield zero AVR with respect to their supplied constraints.

TaskCorrect policy: BrierIncorrect policy: BrierBrier increaseAVR under either policy
S10.0823800.2712080.1888280
S20.1966750.2805800.0839050

Constraint adherence and predictive correctness are assessed separately. The conditional risk guarantee assumes that the true conditional mean belongs to the feasible set. For evidence-state changes, missing or invalid supporting evidence withdraws its relation; available evidence with unknown validity can remain auxiliary. Contradictory active constraints or absence of usable inputs lead to abstention (Appendix Table B.6).

The revised example distinguishes the protocol-supplied mapping from activation conditions (p. 5, lines 221–225) and describes withdrawal and fallback behavior (p. 5, lines 231–234). The existing risk statement gives the feasible-set assumption (p. 6, lines 318–320), and the conclusion summarizes the observed cost of policy misspecification (p. 8, lines 418–419).

AuthorsOpenReview ↗

Addendum: Response to Weakness #4

Following our earlier response, we added a retrospective downstream experiment on UTSW glioma grading to give a concrete example of how a task protocol can specify an authority relation.

Rule specification. WHO CNS5 assigns grades 2 and 3 to IDH-mutant, 1p/19q-codeleted oligodendroglioma. For this experiment, we set p(grade 4)=0p(\mathrm{grade}\ 4)=0 when a curated UTSW record contains both molecular findings, a documented IDH testing method (IHC or NGS), and a prespecified oligodendroglioma eligibility flag derived from the historical tumor type. The clinical classification specifies which grades the molecular evidence can constrain within this scope. This retrospective task conditions on existing molecular reports and historical diagnostic information. Historical tumor types within the prespecified scope, including grade-specific subtypes, map to a single eligibility flag shared by both models. This flag both identifies rule applicability and enters the molecular-report features of both models; the original diagnostic text is excluded from model inputs, and the target grade is withheld from rule compilation. The rule activates for 58 of 618 labeled patients. Five additional molecular candidates stay in the cohort with this rule inactive because their historical types fall outside the prespecified scope. This is a specification for this grading task. Applying the framework to another target would require an evidence-to-constraint mapping for that target. As explained in our earlier response, with usable inputs and no active authority relation, ACF returns its unconstrained prediction. [1, 2]

A further example of a guideline-defined output relation is the assignment of CNS WHO grade 4 to IDH-mutant astrocytoma with documented homozygous deletion of CDKN2A/B, as recommended by [3].

Both fusion models receive the same evidence, eligibility flag, and output projection. ACF additionally protects the active report state from MRI updates, while MRI remains available through auxiliary states and the grading head.

Table 1: Five-fold out-of-fold UTSW grading results (618 patients). Brier is dimension-normalized; AVR is computed over the 58 activated relations. Lower Brier and AVR and higher accuracy are better.

MethodBrierAccuracyAVR
Label prior0.162368.0%1
Label prior + projection0.140674.0%0
Molecular + projection0.094479.6%0
Learned-joint + projection0.089280.6%0
ACF0.092879.3%0

The clinical specification supplies a concrete class exclusion for this task, and both fusion models satisfy all 58 active constraints. Learned-joint has higher observed accuracy and lower Brier than ACF; the paired 95% interval for the Brier difference (ACF minus learned-joint) is [−0.000035, 0.00725][-0.000035,\,0.00725]. Appendix C reports the activation protocol and full results (PDF p. 32, lines 1682–1692 and 1713–1719).

The revised manuscript presents the downstream experiment in Section 4.4 (PDF p. 8, lines 416–431) and the paired case in Figure 4 (p. 9, caption lines 469–473).

References

[1] Louis et al. (2021). The 2021 WHO Classification of Tumors of the Central Nervous System: a summary. Neuro-Oncology, 23(8), 1231–1251.

[2] Reddy et al. (2026). The University of Texas Southwestern Glioma Dataset — MRI, Molecular Markers and Segmentations. Scientific Data, 13, 934.

[3] Brat et al. (2020). cIMPACT-NOW update 5: recommended grading criteria and terminologies for IDH-mutant astrocytomas. Acta Neuropathologica.

Thank you for the rebuttal, I will maintain my positive scores.

Reviewer_uFnPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Authority-Consistent Fusion (ACF), which treats the authority of an evidence source as an externally specified, task-dependent variable. ACF compiles authority rules into graph restrictions and output constraints, protects designated representations from unauthorized inputs, and projects predictions onto a feasible set. The evaluation combines synthetic authority tasks with UTSW MRI-to-semantic retrieval experiments.

Critical strengths
  1. The distinction between authority and reliability is useful. Explicitly specifying which source may constrain a particular decision makes a requirement visible that ordinary fusion weights do not capture.
  2. The evaluation avoids several misleading comparisons. Strong baselines receive the same authority information, and the paper separates graph intervention effects from gains obtained after retraining. It also reports the harm caused by incorrect specifications.
Critical weaknesses
  1. The justification for protected computation is incomplete. Authority over an output relation does not necessarily require isolating the corresponding representation from auxiliary evidence. Clinical context may help interpret an authoritative observation without overriding its decision constraint. APGM prohibits such inputs from affecting protected states. Although auxiliary branches and the final head retain some interaction, the paper does not establish whether these pathways preserve the necessary complementary information. Learned-joint fusion already achieves zero violations with nearly identical clean Brier scores (Table B.3).
  2. The evaluation does not establish that useful authority constraints can be specified for real clinical evidence. The framework assumes externally supplied policies, validity states, and decision relations. Yet a valid observation does not automatically justify a hard class exclusion or probability-order constraint. UTSW evaluates geometry and graph communication rather than this specification process. The authority tasks are synthetic. Table B.5 shows why this matters: incorrect policies substantially worsen prediction while preserving zero violations. The conditional risk guarantee assumes that the true conditional mean belongs to the feasible set; the real-data experiments do not substantiate this premise.
Questions for authors
  1. What justifies translating authority over an output relation into restrictions on representation updates? Can a controlled comparison show whether APGM preserves useful cross-modal interactions relative to learned-joint fusion with the same output constraints?
  2. Can you provide a concrete clinical specification linking evidence validity to the proposed hard constraints, and evaluate whether those constraints remain appropriate when real evidence sources disagree?
5 comments
AuthorsOpenReview ↗

Response to Reviewer uFnP-Part1

Thank you for recognizing the distinction between authority and reliability and the use of information-matched comparisons. Your questions identify two requirements that deserve a clearer separation: the purpose of protected evidence states and the justification of the supplied decision constraints.

All page and line references below refer to the revised manuscript PDF.

1. Protected computation and complementary information

APGM serves an additional source-dependency audit requirement alongside output compliance. Its practical role is to preserve a designated reference state tied to the protocol-permitted evidence while the final prediction uses auxiliary context. During an implementation audit, one can fix the authorized evidence, parameters, and compilation, change the auxiliary inputs, and inspect the designated state before readout. A state change reveals a dependency across the declared boundary even if the projected output remains admissible. A task requiring output compliance alone can use learned-joint fusion with the same projection.

The evaluated architecture constructs the protected source state from its local encoding, source-local metadata, and read-only supports. Auxiliary nodes can read the designated source, and the head receives all final node representations, averaged across relation branches, together with the policy embedding. These pathways allow context to influence the final prediction. Their predictive utility is tested below; the designated-state restriction is a separate task requirement.

For the requested controlled comparison, we reorganized saved S1 evaluations of ACF and learned-joint. The variants use matched source encoders, graph-update architecture, training configurations, and the same output projection and loss; their dependency masks implement the different protection requirements. Each model is trained separately. Paired interventions hold the external policy and authorized inputs fixed while changing auxiliary evidence.

Table 1: Auxiliary predictive utility and designated-state invariance under matched S1 interventions. Both methods have AVR = 0 in every row. Lower residual Brier indicates better prediction of the binary choice left open by the class constraint.

Auxiliary conditionResidual Brier: ACFResidual Brier: learned-jointMean state shift: ACFMean state shift: learned-joint
rb=0.60r_b=0.600.2813430.28080900.308086
rb=0.80r_b=0.800.1646190.16475400.062747
rb=0.95r_b=0.950.0761330.07415300.251792
Conflict strength 80.3088530.28860101.700822

Values average three model initializations within each of five independent data seeds. Residual Brier measures the remaining binary choice. The reliability rows use η=0\eta=0 and κ=0\kappa=0; the conflict row uses η=0\eta=0, rb=0.80r_b=0.80, κ=1\kappa=1, and strength = 8. State shifts compare each trained model with its paired clean reference (η=0\eta=0, rb=0.80r_b=0.80, κ=0.1\kappa=0.1, strength = 1), holding the external policy and authorized inputs fixed. Thus, the rb=0.80r_b=0.80 row also changes κ\kappa from 0.1 to 0. Each shift is the mean within-model ℓ∞\ell_\infty change in the corresponding relation-designated state; 0 means that the state remains unchanged. Shift magnitudes use each model's own representation coordinates.

ACF retains auxiliary predictive utility in the tested reliability settings: residual Brier improves from 0.281343 to 0.076133, while the designated state remains unchanged. Learned-joint shows a similar predictive response. In the strongest tested conflict condition, learned-joint has lower mean residual Brier (0.288601 versus 0.308853). These comparisons measure useful auxiliary prediction and designated-state invariance separately. Whether the same pathways preserve context essential to interpreting a clinical observation requires a task-specific evaluation.

The revised Section 3.1 states the additional requirement (pp. 4–5, lines 214–219), and the Illustrative example specifies the permitted dependencies and final readout (p. 5, lines 221–234). The Introduction and formal task statement use the same scope (p. 3, lines 122–132; p. 5, lines 256–258). Table 1 above presents the controlled comparison in this response.

AuthorsOpenReview ↗

Response to Reviewer uFnP-Part2

2. Clinical specifications, validity, and disagreement

The concrete specification and disagreement tests supplied by the present study are those of AIS. The requested clinically validated specification and evaluation under real-source disagreement remain outside the available evidence. UTSW supports the evaluated representation and communication choices through MRI-to-semantic retrieval.

The evidence-to-constraint mapping is supplied by the task protocol; validity and availability govern its activation. In the four-class example, the protocol maps a source statement to the permitted classes when its supporting evidence is valid and available. AIS supplies this mapping through known task construction. For clinical use, the mapping itself would require justification for the specified target, evidence statement, and exclusion or probability order, including any numerical margin.

The existing experiments quantify why that distinction matters. With an incorrect policy, ACF Brier rises from 0.082380 to 0.271208 on S1 and from 0.196675 to 0.280580 on S2 while AVR remains 0 (Appendix Table B.5). The conditional risk guarantee requires the true conditional mean to belong to the feasible set. Evidence withdrawal and abstention address validity, availability, and incompatible constraints (Appendix Table B.6); predictive consequences of a feasible but incorrect policy are measured separately by the misspecification experiment.

The revised example makes the mapping and activation conditions explicit (p. 5, lines 221–225), and Section 3.1 identifies the evidence supplied by AIS and UTSW (p. 5, lines 235–237). The conclusion states the clinical validation requirement (p. 8, lines 421–423); the risk assumption remains explicit in Section 3.5 (p. 6, lines 318–320).

AuthorsOpenReview ↗

Addendum: Response to Weakness #2

Following our earlier response, we added a retrospective UTSW grading comparison that measures predictive performance and report-state protection under the same clinical output constraint.

Protected computation with matched projection. ACF preserves a report-derived state for auditing by allowing the active report node to read only itself and read-only supports. MRI still contributes through auxiliary states and the grading head, including to the remaining choice between grades 2 and 3. Learned-joint also permits MRI inputs to update the report state. Over five held-out patient folds, ACF and learned-joint achieve 79.3% and 80.6% accuracy and 0.0928 and 0.0892 Brier, respectively. Both satisfy all 58 active constraints. With the molecular report and rule fixed, MRI noise and occlusion produce maximum within-model report-state shifts of 0 and 0 for ACF, compared with 1.698 and 0.478 for learned-joint. Learned-joint has higher observed accuracy and lower Brier; ACF maintains the designated state under these fixed-report interventions. The comparison evaluates the additional audit requirement separately from predictive performance.

Clinical specification. WHO CNS5 assigns grades 2 and 3 to IDH-mutant, 1p/19q-codeleted oligodendroglioma. We operationalize this as p(grade 4)=0p(\mathrm{grade}\ 4)=0 when a curated UTSW record contains both molecular findings, a documented IDH testing method (IHC or NGS), and a prespecified oligodendroglioma eligibility flag derived from the historical tumor type. The clinical classification specifies which grades the molecular evidence can constrain within this scope. This retrospective task conditions on existing molecular reports and historical diagnostic information. Historical tumor types within the prespecified scope, including grade-specific subtypes, map to a single eligibility flag shared by both models. This flag both identifies rule applicability and enters the molecular-report features of both models; the original diagnostic text is excluded from model inputs, and the target grade is withheld from rule compilation. The rule activates for 58 of 618 labeled patients, all with a recorded grade in {2,3}\{2,3\}. The protocol accepts curated records, with assay-level validity recorded as unreported. [1, 2]

As a further illustration of guideline-defined output relations, the ASCO/CAP testing guideline classifies an interpretable HER2 IHC 3+ result as positive in breast cancer, while IHC 2+ requires further testing under the algorithm [3].

Observed disagreement. Before projection, grade 4 was the highest-probability grade for 16 of the 58 active cases with ACF and 4 of the 58 with learned-joint. Projection enforces the supplied rule in these cases. This comparison captures disagreement between preliminary model predictions and the molecular rule. Assessing whether the rule remains appropriate when clinical assays disagree would require paired assay data.

See Section 4.4 for the matched comparison (PDF p. 8, lines 416–431), Figure 4 for the paired case (p. 9, caption lines 469–473), and Appendix C for the rule and disagreement analysis (p. 32, lines 1682–1692 and 1701–1705).

References

[1] Louis et al. (2021). The 2021 WHO Classification of Tumors of the Central Nervous System: a summary. Neuro-Oncology, 23(8), 1231–1251.

[2] Reddy et al. (2026). The University of Texas Southwestern Glioma Dataset — MRI, Molecular Markers and Segmentations. Scientific Data, 13, 934.

[3] Wolff et al. (2023). Human Epidermal Growth Factor Receptor 2 Testing in Breast Cancer: ASCO-CAP Guideline Update. Journal of Clinical Oncology.

Reviewer_hJnzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates authority-asymmetric multimodal learning by distinguishing evidential authority from predictive utility and source reliability. The authors propose Authority-Consistent Fusion (ACF), a framework consisting of three components: an Authority Specification Compiler (ASC), which translates externally specified authority policies into admissible graph dependencies and output constraints; an Authority-Protected Graph Mixer (APGM), which restricts unauthorized information propagation into protected representations; and an Authority-Consistent Projection (ACP), which enforces activated decision constraints through projection onto a feasible output set. The paper presents theoretical results concerning constraint feasibility, computational non-interference, projection properties, and conditional squared-loss guarantees. The experimental evaluation consists of UTSW MRI-to-semantic retrieval experiments, which examine the contributions of SPD representations, graph communication, and semantic anchors, and the controlled Authority Identification Suite (AIS), which evaluates authority compliance, reliability interventions, and incorrect authority specifications. The results indicate that ACF can enforce the specified authority constraints while maintaining predictive performance comparable to information-matched fusion baselines.

Critical strengths

The paper addresses an interesting and relevant problem by explicitly distinguishing evidential authority from predictive utility and reliability in multimodal learning. This distinction provides a useful perspective for applications in which different evidence sources may have different roles in constraining particular decision relations. Rather than relying exclusively on learned modality weights or reliability estimates, the proposed framework explicitly represents which sources are permitted to constrain specific decisions. The framework is technically coherent. In particular, the separation between computational non-interference and final-output constraint satisfaction is meaningful. APGM restricts unauthorized information dependencies in protected representations, while ACP ensures that the final predictions satisfy the activated constraints. This distinction provides a principled formulation of two different aspects of authority consistency. Another strength is the substantial effort devoted to theoretical development and empirical evaluation. The paper presents a systematic analysis of its guarantees under explicit assumptions, and the extensive appendix contains useful proofs, experimental details, and additional analyses. The controlled AIS experiments use information-matched baselines and interventions to distinguish authority from reliability and predictive utility. The experiments involving incorrect authority specifications are particularly informative, as they illustrate that constraint compliance does not necessarily imply predictive correctness. The UTSW experiments also provide useful component-level comparisons regarding representation geometry and graph communication. Overall, the paper presents a meaningful problem formulation and a coherent technical framework, supported by substantial theoretical and empirical work.

Critical weaknesses
  1. The main paper does not sufficiently communicate the core methodology despite the substantial amount of work presented. The authors have clearly invested considerable effort in developing the framework, theoretical analysis, and experimental evaluation. The extensive appendix contains many useful details and demonstrates the breadth of the investigation. However, I find it somewhat unfortunate that the main paper does not communicate the central methodology with the same level of clarity and completeness. I understand that conference papers are subject to strict page limitations and that detailed proofs, additional experiments, and implementation details are often appropriately placed in the appendix. Nevertheless, the main paper should remain sufficiently self-contained for readers to understand the essential operation and technical contribution of the proposed method. In particular, the descriptions of ASC and APGM remain relatively abstract. The ASC formulation specifies a mapping from an authority specification to graph dependencies and output constraints, but the main text would benefit from a concrete example illustrating how this compilation is performed. Similarly, although APGM is formulated through restricted message passing, the construction of relation-specific protected representations, the corresponding computational dependencies, and their integration into the prediction process are not explained sufficiently clearly. These aspects concern the central mechanisms of the proposed method rather than merely secondary implementation details. Consequently, readers must rely heavily on the supplementary material to reconstruct the overall computational procedure and fully assess the methodological contribution. I believe the paper contains substantial technical work, but its contribution would be communicated much more effectively if the main text devoted greater attention to explaining the core method. A concise end-to-end example and a clearer architectural description would significantly strengthen the paper.
  2. The technical contribution of the combined framework could be distinguished more clearly from established constrained-learning approaches. The proposed framework combines externally specified authority rules, restricted graph communication, and projection onto a convex feasible set. These components are technically reasonable and form a coherent overall design. However, the individual mechanisms have clear connections to existing techniques in constrained learning, masked message passing, and convex optimization. The principal contribution therefore appears to lie in their integration under an authority-conditioned formulation, particularly the distinction between computational and decision-level compliance. Such an integration can constitute a meaningful methodological contribution, and I do not consider the use of established mathematical tools a weakness by itself. Nevertheless, the paper would benefit from a more explicit explanation of what is gained from combining these components and how the resulting framework differs from conventional constrained fusion or projection-based prediction. This clarification would help readers better identify the specific technical contribution of ACF.
  3. The additional role of protected graph computation beyond final-output projection could be better motivated. The separation between APGM and ACP is theoretically meaningful, since satisfying final-output constraints does not necessarily imply that protected intermediate representations are unaffected by unauthorized inputs. However, the experimental results also show that projection-based, information-matched baselines can satisfy the activated output constraints. For example, the learned-joint baseline achieves zero authority-violation rate in the reported clean settings, while ACF provides the additional guarantee of protected-state non-interference. The controlled intervention experiments support this computational protection property. Nevertheless, the paper could better explain why this additional guarantee is important for the intended use cases and how its value should be interpreted beyond final-output compliance. I do not necessarily expect a large additional experiment. A clearer technical discussion, supported by an illustrative failure case for output-only protection, would help establish the motivation for APGM and clarify the distinct roles of the two modules.
  4. The relationship between the two experimental settings and the overall claims could be explained more explicitly. The UTSW experiments primarily investigate representation geometry, graph communication, and semantic anchors, whereas the AIS experiments examine the authority-specific mechanisms. Both evaluations provide useful evidence, and their separation allows different aspects of the framework to be analyzed under controlled conditions. However, the main paper could better distinguish the conclusions supported by these two experimental settings. In particular, improvements in UTSW retrieval performance primarily demonstrate the effectiveness of the evaluated fusion backbone, whereas the AIS experiments provide more direct evidence concerning authority compliance and computational protection. A more focused discussion of this distinction would make the empirical contribution easier to interpret and would help avoid conflating improvements in general multimodal representation quality with the guarantees provided by the authority-specific components. Overall, my concerns primarily relate to the clarity of the core methodology, the identification of the specific technical contribution, and the interpretation of the experimental evidence. They do not fundamentally undermine the internal consistency of the proposed formulation.
Questions for authors
  1. Clarity of the core framework: Could the authors provide a concise, end-to-end example in the main paper showing how an authority specification (A_q) is compiled into the admissible graph (G_q) and output constraints (B_qp \succeq b_q), how APGM constructs and updates protected representations, and how ACP produces the final prediction? Such an example would substantially improve the accessibility and technical clarity of the method.
  2. Distinction between the roles of APGM and ACP: Since output projection already ensures satisfaction of activated decision constraints, could the authors clarify the additional technical motivation and intended significance of computational non-interference? In particular, an illustrative example of a limitation of output-only protection would help readers understand why both components are included.
  3. Specific methodological contribution: Could the authors explain more explicitly which capabilities arise from the joint ASC–APGM–ACP formulation, particularly compared with conventional constrained-learning or projection-based fusion approaches? This would help clarify the novelty of the proposed integration without requiring each individual component to be entirely new.
  4. Interpretation of the experimental evidence: Could the authors more clearly distinguish the findings supported by the UTSW geometric-fusion experiments from those supported by the AIS authority-mechanism experiments? This would help establish a more precise correspondence between the reported results and the paper's central claims.
4 comments
AuthorsOpenReview ↗

Response to Reviewer hJnz- Part 1

Thank you for the careful reading and the suggestions on presentation, methodological scope, and evidence. We have prepared a main-text revision centered on an illustrative example, an explicit source-auditing requirement, and a direct correspondence between claims and experiments. We address the four questions below.

All page and line references below refer to the revised manuscript PDF.

Q1. End-to-end operation of ASC, APGM, and ACP

ASC translates a task specification and the current evidence state into relation-specific dependencies and output constraints. APGM constructs the designated evidence states under these dependencies, while auxiliary branches combine complementary information. The head reads the resulting states, and ACP projects its prediction onto the active feasible set.

Consider a four-class task with one active relation, source aa, three auxiliary sources, and read-only support nodes. The task protocol maps aa's statement to the permitted classes {1,2}\{1,2\} when aa is valid and available, and separately requires a source-auditable state for aa. ASC compiles Cq={p∈Δ3:p3=p4=0}\mathcal{C}_q=\{\mathbf{p}\in\Delta^3:p_3=p_4=0\}, represented by the active constraint rows −e3⊤-\mathbf{e}_3^{\top} and −e4⊤-\mathbf{e}_4^{\top} with zero right-hand sides. In the compiled graph, aa reads itself and the supports; auxiliary nodes may read all usable sources and supports. APGM initializes each relation branch with source encodings and source-local metadata, then applies shared updates with attention restricted to the permitted neighborhood. Supports remain fixed during the forward pass. The head averages each node across relation branches (one here) and concatenates the node vectors with the policy embedding. If it yields p0=(0.1,0.2,0.3,0.4)\mathbf{p}_0=(0.1,0.2,0.3,0.4), ACP's Euclidean projection is p⋆=(0.45,0.55,0,0)\mathbf{p}^{\star}=(0.45,0.55,0,0). Auxiliary inputs can change the probabilities within {1,2}\{1,2\}; with parameters, compilation, and authorized inputs fixed, aa's pre-readout state remains invariant to those changes. If aa becomes invalid or unavailable, ASC withdraws this relation and recompiles the graph and constraints. With usable inputs and no active constraints, ACF returns p0\mathbf{p}_0; incompatible constraints or absence of usable inputs leads to abstention.

The revised Section 3.1 states the compiler inputs and the roles of ASC, APGM, and ACP (pp. 4–5, lines 214–219), followed by the Illustrative example covering branch initialization, permitted dependencies, final readout, projection, and fallback behavior (p. 5, lines 221–234).

AuthorsOpenReview ↗

Response to Reviewer hJnz- Part2

Q2. The additional role and intended value of APGM

APGM provides a stable reference state for auditing declared source dependencies while the final prediction uses auxiliary context. In an implementation audit, one can hold the authorized evidence, parameters, and compiled specification fixed, vary the auxiliary inputs, and inspect the designated state before readout. A change in that state reveals an auxiliary dependency across the declared boundary, even when the final prediction satisfies every output constraint. APGM enforces this dependency requirement; ACP enforces the output relations. Tasks requiring output compliance alone can use projected joint fusion.

Appendix Table B.8 illustrates this audit: a deliberately inserted auxiliary bypass produces mean state shifts of 0.062627/0.056688 on S1/S2 while AVR remains 0. The state audit detects the changed dependency even though the output-compliance check still passes.

To examine auxiliary predictive utility alongside this property, we reorganized saved S1 evaluations of separately trained ACF and learned-joint models with matched encoder and graph-update architectures, training configurations, and output projection.

Table 1: Auxiliary predictive utility and relation-designated state shifts on S1. Both methods use the same output projection and have AVR = 0 in every condition.

Auxiliary conditionResidual Brier, ACFResidual Brier, learned-jointMean state shift, ACFMean state shift, learned-joint
rb=0.60r_b = 0.600.2813430.28080900.308086
rb=0.80r_b = 0.800.1646190.16475400.062747
rb=0.95r_b = 0.950.0761330.07415300.251792
Conflict strength 80.3088530.28860101.700822

Values average three model initializations within each of five independent data seeds. Residual Brier measures the remaining binary choice. The reliability rows use η=0\eta=0 and κ=0\kappa=0; the conflict row uses η=0\eta=0, rb=0.80r_b=0.80, κ=1\kappa=1, and strength = 8. State shifts compare each trained model with its paired clean reference (η=0\eta=0, rb=0.80r_b=0.80, κ=0.1\kappa=0.1, strength = 1), holding the external policy and authorized inputs fixed. Thus, the rb=0.80r_b=0.80 row also changes κ\kappa from 0.1 to 0. Each shift is the mean within-model ℓ∞\ell_\infty change in the corresponding relation-designated state; 0 means that the state remains unchanged. Shift magnitudes use each model's own representation coordinates.

ACF retains auxiliary predictive utility in the tested reliability settings: residual Brier decreases from 0.281343 to 0.076133 while its designated state remains unchanged. Learned-joint shows a similar predictive response. In the strongest tested conflict condition, learned-joint has lower mean residual Brier (0.288601 versus 0.308853). These results separate the source-dependency property from predictive performance under each intervention.

The revised Overview states the audit requirement (pp. 4–5, lines 214–219), with matching scope in the Introduction (p. 3, lines 122–132) and Section 3.2 (p. 5, lines 256–258). Table 3 identifies the deliberate-bypass audit and the valid-APGM reference of 0 (p. 8, lines 388–389). Table 1 above presents the matched comparison in this response.

AuthorsOpenReview ↗

Response to Reviewer hJnz- Part3

Q3. Contribution of the joint formulation

The joint formulation makes the same task specification govern both the construction of designated evidence states and the admissible final prediction. Its contribution is the coordination of these two requirements through relation-specific source assignments, validity, and availability.

The existing comparisons identify what each mechanism contributes. Graph-only fusion has S1 AVR = 1; applying projection to the same checkpoint reduces it to 0 (Appendix Table B.3). Projected learned-joint fusion has AVR = 0 but its designated states change under the auxiliary interventions in Table 1 above. ACF has both AVR = 0 and designated-state shift = 0 in those comparisons. ASC coordinates the graph restrictions and output constraints, including their update when supporting evidence is withdrawn. Auxiliary branches remain available to the final head.

Section 3.1 states this integration (pp. 4–5, lines 214–219) and illustrates the two compiled objects and their coordinated update (p. 5, lines 221–234). The conditional risk statement separately specifies the assumption that the true conditional mean belongs to the feasible set (p. 6, lines 318–320).

Q4. Interpretation of AIS and UTSW

AIS evaluates authority-specific mechanisms under controlled specifications; UTSW evaluates the geometric and communication choices of the fusion backbone. Each setting supports a distinct set of claims.

On AIS, mean clean Brier scores for ACF/learned-joint are 0.082380/0.082434 on S1 and 0.196675/0.196741 on S2 (Appendix Table B.3). The intervention and failure analyses evaluate output adherence, state invariance, and specification sensitivity. For example, policy misspecification raises ACF Brier to 0.271208/0.280580 while AVR remains zero (Appendix Table B.5). On UTSW, the matched SPD-versus-Euclidean comparison yields +6.485 mAP points, and uniform-versus-identity communication yields +2.482 points (Appendix Table B.12). These are conditional backbone comparisons. Applying authority constraints clinically additionally requires independently validated evidence-to-constraint specifications.

The revised abstract attributes the Brier results to AIS and reports the matched UTSW contrasts separately (p. 1, lines 024–029). Section 3.1 states the AIS/UTSW distinction (p. 5, lines 235–237), and the Table 2 caption identifies matched configuration contrasts (p. 7, lines 334–335). The clean-Brier comparison is specified in the Introduction (p. 3, lines 145–147), Table 3 (p. 8, lines 382–384), and the conclusion (p. 8, lines 416–418). The clinical validation requirement appears in the conclusion (p. 8, lines 421–423).

AuthorsOpenReview ↗

General response

Thank you again for your thoughtful review. As the discussion period draws to a close, we would appreciate your feedback on whether our response addresses the points you raised. If any part of our response needs clarification, we would be happy to provide it. Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.