GraphSelect for Budgeted Representation Selection in Multimodal Graph Inference

Multimodal graphs describe connected entities through relations, text, and images. Predictors that combine these sources support recommendation, retrieval, and node classification, but their encoded inputs can exceed the cache, transfer queue, or accelerator space available for a later inference stage. When the graph and trained predictor remain unchanged, the system must decide which node text and image representations to retain under separate capacity limits. We formulate this decision as budgeted representation selection. Given an encoded graph, candidate node representations, and two capacities, the task selects a joint subset to preserve the classes predicted when every candidate is available. Graph propagation makes each contribution depend on the neighboring and cross modal representations already retained, while existing attribution methods usually rank candidates at a single input state. Exact studies on six graphs show that deletion scores can miss the best addition, prediction agreement often stops distinguishing candidates, and reference state gains become stale as the subset grows. Guided by these results, GraphSelect first fills both modality budgets with exact addition gains. It then recomputes removal and addition scores for the current subset and tests promising within modality exchanges with the unchanged predictor. Under a shared selection protocol, GraphSelect achieves lower selection loss than six published attribution and explanation procedures. Tests on nine trained graph and multimodal architectures further show that increasing capacity moves recovery and downstream predictions toward the corresponding full input result. At 20% capacity per modality, the mean accuracy gap is 0.10 percentage points. Together, these results show that GraphSelect selects a small joint input and yields predictions close to those of the same trained model with complete candidate input.