- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether a specified graph constructor should be invoked for a multimodal dataset and task. MAG-SCOUT collects a bounded relational sketch, combines necessity, constructability and utility diagnostics, and predicts the performance gain over a graph-free pipeline. Predicted cost and calibrated uncertainty enter a Build/Skip score. Evaluation includes nine constructors on six benchmarks, prediction and policy evaluation on a 13-parent-source registry, and component ablations on a separate rebuilt 11-source bank. At a validation-selected 95% retention target, the paper reports 96.7% task-macro retention of positive gain and 23.6% avoided graph work.
Critical strengths
-
Equations 2–3 distinguish the incremental benefit over raw multimodal learning from absolute graph-model performance and account for construction cost. Table 1 reports both positive and negative gains under the evaluated pipelines. This motivates a practical assessment stage before committing to graph construction.
-
Appendix B.2 establishes the greedy approximation for regularised log-det selection with fixed features and a cardinality budget, while explicitly limiting the guarantee to sketch volume. Appendix B.4 separates fitting, scale calibration, policy validation and outer testing by parent source. These are useful foundations for assessing the method without conflating the sketch guarantee with downstream predictive reliability.
Critical weaknesses
-
The central decision and uncertainty results need reconciliation. At a 95% retention floor, Figure 4(b) places MAG-SCOUT's avoided graph work at approximately 33%, whereas Section 6.4 and Table 9 report 23.6%. At the 90% floor, the corresponding values are approximately 35% and 24.2%. These differences change the apparent comparison with competing policies and require corrected numerical inputs and plotting scripts.
Table 6 also needs a more precise calibration account. Every reported mean width falls to approximately 78% of its raw value, while every coverage increases; for NC at the 80% nominal level, width falls from 4.69 to 3.66 and coverage rises from 72.5% to 77.4%. With fixed means and one common multiplier below one, each calibrated interval is contained in its raw interval, so coverage cannot increase on the same observations. Aggregation over outer folds with different multipliers can produce narrower mean widths and higher mean coverage, however. The aggregate table alone therefore does not establish a contradiction; fold-specific multipliers and paired predictions are needed to assess compatibility with Equation 41.
-
The released implementation does not substantiate the proposed training procedure or benchmark comparisons. In the repository snapshot archived on 19 October 2026,
experiments/evaluate_q1.pygenerates synthetic targets and defines MAG-SCOUT predictions as those targets plus noise.evaluate_q3_policy.pylikewise generates gains and costs randomly, andevaluate_q4_q5.pyprints a stability conclusion without running a sweep. These demonstrations do not reproduce the reported source-held-out results.There are also consequential method mismatches.
MAGScout.forwarddoes not call its relation encoder, and the supplied training loop takes an independently generated relation context. The loss omits the Gaussian uncertainty term in Equation 39, leaving the scale head without supervision through that objective. The NumPy selector omits the identity regularisation used by Algorithm 1; a direct check with duplicate candidates selects one relation despite a strictly larger regularised objective at budget two. Its projection update also raises a broadcasting error with three candidates in two dimensions. The one-time refinement ablated in Tables 3 and 8 is not specified as an executable procedure. A connected implementation and the actual experimental records are needed to determine which method produced the results. These observations establish gaps in the released reproduction package; the manuscript's experimental generation process remains unresolved.
Questions for authors
-
Which values in Figure 4(b) or Table 9 are correct? Could the authors provide the corresponding plotting inputs and held-out episode records, including source, task, constructor, realised gain, measured cost, prediction and action? This would clarify the reported policy comparison.
-
Appendix B.4 specifies calibration within each outer-training split. Could the authors provide the multiplier for each fold and seed, paired interval endpoints, and the coverage/width aggregation used for Table 6? What is the scope of "one global scale multiplier", and which uncertainty-training implementation produced these results? This would resolve whether the reported changes follow the stated procedure.
-
Could the authors identify the implementation that connects relation encoding, discrete sketch selection, diagnostic extraction and the four-term loss, and specify one-time refinement? How does the selector used in the experiments implement the regularised marginal in Equation 19? This would distinguish an incomplete release from a mismatch in the evaluated algorithm.
-
Could the authors report realised downstream utility or regret and net computation after assessment overhead, with a paired comparison against NetInfoF? Section 6.4 gives a saved-work difference of 0.5 points with a 95% source-bootstrap interval of [−6.7, 7.8]. Retained positive-gain mass does not account for performance losses on negative-gain Build cases, and Equation 50 does not subtract assessment overhead. These results would clarify the policy's incremental net benefit over NetInfoF.