- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies a specific issue within the mean-field variational inference process of optimal subset (OS) oracle frameworks (such as EquiVSet and INSET): the estimation of the gradient of a multilinear extension via Monte Carlo (MC) sampling. According to the authors, this step is computationally expensive and introduces stochasticity that destabilizes the optimization trajectory. To address this, the authors propose "ReSet," which reinterprets the ELBO as a continuous relaxation of a set function and learns a differentiable surrogate function, , to replace the MC-based gradient estimates with the gradient of . Specifically, the method initializes using a recognition network, performs iterations of gradient ascent, incorporating an entropy gradient and Langevin noise into . And trains the entire framework by minimizing the binary cross-entropy (BCE) loss with respect to the final . The step size and temperature are also treated as learnable parameters.
Critical strengths
1. Consistent performance and efficiency improvements. Both the EquiVSet and INSET frameworks demonstrate higher performance than existing OS oracles, including iDiffMF, on most of the 16+ datasets. At the same time, inference time and memory are reduced.
2. A direct and proven solution to cost issues. Since removing sampling eliminates the evaluation of the per-sample set function, the proposed method directly and logically resolves the computational cost problem among the two issues raised by the authors. Their analysis quantitatively demonstrate this effect by varying the sample size and set size.
Critical weaknesses
1. The claimed optimization instability is not empirically demonstrated, and the paper’s own results contradict it. Instability is shown only in a toy example. Gradient variance and seed-to-seed variance are never measured on real tasks. In Table 11, increasing MC samples from 10 to 50 barely changes performance, and the gap to ReSet remains. This suggests that variance is not the bottleneck. The large gains are therefore more likely due to the change in model class than to more stable gradients. No control experiment separates these two explanations.
2. The theoretical framing does not match the implemented algorithm. is trained only with BCE on and is not trained to approximate any multilinear extension. The optimized objective is therefore not the ELBO of . With in all main experiments, the method reduces to a feed-forward predictor. Thm 3.1 relies on -weak DR-submodularity, which is neither enforced nor measured, and it analyzes continuous greedy from rather than the actual RecNet-initialized update. The learnable temperature changes by less than 1% during training, so the claimed adaptive trade-off is not supported.
Questions for authors
-
Demonstration of instability. Are there actual results comparing the gradient variance of MC estimation and ReSET, and the variance of the learning curve between seeds, in the assignment (e.g., Toys, Two Moons)? Also, how can the finding in Table 11 that increasing the number of MC samples barely improves performance be reconciled with the claim that "stochasticity induces a suboptimal solution"?
-
Isolation of the cause of improvement. Is there a comparison between defining using the same architecture as ReSet (including MHA) and estimating the multilinear gradient using the traditional method via MC? Please clarify whether "EquiVSet-MHA" in Table 8 is this setting. This comparison is "learned relaxation vs. I believe it is the only way to separate the MC's effect.
-
Source of the sharp increase. What caused the achievement of 99.7 in Two Moons, where all baselines remained around 58–59? The same question is also asked for Furniture (17→36, 280 training samples). Can these datasets demonstrate the MHA-augmented baseline and the results?
2 comments
Looking Forward to Your Feedback
Dear Reviewer 39UD,
Thank you again for your detailed feedback. We have added variance measurements (Appendix F.5), controlled ablations (Appendix F.2), and a comparison of versus to examine the role of the optimization step (Appendix F.6).
As the Author–Reviewer Discussion period draws to a close, we would appreciate your thoughts on these results and the clarifications in our response. We would be happy to discuss any concerns that remain.
Best regards,
The Authors
Response to Reviewer 39UD
We thank the reviewer for their feedback and respond point by point below. All newly added or revised content is highlighted in the revised manuscript.
Original Table 11 (now Table 12) does not support the blanket claim that more MC samples barely improve performance: increasing samples from 10 to 50 raises EquiVSet’s MJC from 71.6 to 75.9 (+4.3 percentage points), although INSET gains only 0.4 points.
We added measurements on Toys in Appendix F.5 (Table 13): increasing samples from 10 to 50 reduces gradient variance by approximately 80%, including for MHA baselines.
ReSet eliminates MC sampling variance by directly differentiating the learned relaxation. We also measured per-epoch validation MJC variance across training seeds on Toys. Figure 8 shows that, after an initial peak, EquiVSet-R and INSET-R have 53.0% and 7.9% lower mean variance than their baselines during the second half of training. Lower gradient variance need not proportionally improve MJC, as the final top-k selection may remain unchanged.
ReSet backpropagates BCE through unrolled refinement for optimal-subset prediction, avoiding costly explicit multilinear-matching targets—a core motivation. The objective is thus a learned variational surrogate, not an assumed exact ELBO of pθ(S∣V). The original Appendix F.1 and Figure 7 already demonstrate strong alignment with the induced multilinear extension’s gradients on synthetic inputs: median cosine similarities are 0.845–0.952 for EquiVSet-R and 0.944–0.982 for INSET-R.
The original -MHA baselines already matched their ReSet counterparts’ architectures. New ablations in Appendix F.2 (Table 9) isolate MC estimation versus learned relaxation by controlling other factors, showing higher mean MJC on all six datasets in both families. See our response to Reviewer 94Ud.
A plausible explanation for the sharp gains is that ReSet learns a set objective tailored to finite-step inference and removes MC gradient noise. Improved inclusion-probability rankings may then yield substantial MJC gains after discrete selection.
Calling K=1 merely feed-forward overlooks its variational optimization stage. As in EquiVSet and INSET, recognition initializes a subset distribution, which optimization adapts to the learned set objective. In contrast, K=0 returns recognition marginals directly—a restricted DeepSet-type predictor in our implementation.
The new comparison in Appendix F.6 (Table 14) shows gains on all six datasets in both families. Dataset-averaged MJC (%) is:
These gains demonstrate the substantial benefit of one variational optimization step. Both EquiVSet and INSET use K=1 by default, supporting this choice for amortized inference and ensuring fair comparison.
Structural assumptions. Appendix F.8 (Tables 17–18) presents two surrogate implementations guaranteeing γ=L=1 by construction, with numerical verification. Their measured mean normalized approximation ratios are 0.9875 and 0.9922 across two synthetic and six real datasets. See our response to Reviewer eZVm.
RecNet initialization. Zero initialization follows standard continuous greedy and avoids dependence on initialization quality. We extend the approximation and refinement analyses to feasible RecNet initialization in Appendices A.1 and A.4, respectively, establishing compatibility with our framework while accounting for the initial point in the approximation bound.
The original temperature’s limited change may mainly reflect the small shared learning rate and its decay. We added a relative-temperature strategy in Appendix F.6: T(V)=λs(V), where s(V) accounts for energy/entropy gradient scales and positive λ is learned through the original supervision with a separate learning rate. We measure the effective trade-off by r(V)=∥T(V)∇H∥2/∥∇G∥2. Across both families and six product datasets, Table 16 shows relative changes in mean r from −76.61% to +179.53%, exceeding 10% in magnitude in nine of twelve comparisons. Average MJC gains over fixed T=1 are 0.18 and 1.49 percentage points for EquiVSet-R and INSET-R, respectively, supporting adaptive energy–entropy balancing with comparable or improved performance.