- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces FARE, a method that constructs scenario fingerprints based on normalized probe scores. It then employs k-medoids clustering to select representative scenarios, thereby reducing the evaluation cost during LLM-based heuristic discover. The authors validate this approach within the context of 5G scheduling and ECN marking, reporting superior overall performance compared to alternative scenario-selection strategies.
Critical strengths
-
Rather than selecting scenarios only to preserve the ranking of probe programs, FARE focuses on whether different scenarios capture distinct program behaviors. The experiments show that the Order oracle can closely preserve the full-pool ranking of probe programs while still failing to cover some scenario clusters.
-
The authors validate their method on a full-stack 5G emulation platform and report both the one-time fingerprinting cost and the reduction in candidate evaluation cost, rather than relying only on offline clustering metrics.
Critical weaknesses
-
The 'Program Disagreement Distance' (PDD) metric measures the mean absolute difference in normalized probe scores between scenarios, rather than changes in the relative performance of probe programs. If all probe programs score lower by a similar amount on a harder scenario, PDD may be large even when their relative ranking remains unchanged. Conversely, even a small change in scores could lead to a reversal in the relative ranking of programs. As a result, FARE's clustering objective may emphasize differences in scenario difficulty rather than differences in relative program performance.
-
The upper bound established in Proposition 1 depends on program-specific Lipschitz constants and approximation errors. For the scheduling task, the reported Lipschitz scale and representation error give , which already exceeds the maximum possible score difference of 1, even before adding the term. Thus, the resulting bound is trivial under these values. Furthermore, estimating for newly generated programs appears to require full-pool evaluation, making the bound difficult to use during search to assess how well the selected scenarios represent the pool.
Questions for authors
-
Why does PDD use absolute differences in normalized probe scores without separating scenario difficulty from differences in relative probe performance?
-
Beyond the leave-one-probe-out test in Appendix F.2, how does FARE handle highly correlated or redundant probe programs? If the proportions of different probe families change, does the selection of representative scenarios remain stable?
-
Could you provide a non-trivial error bound for Proposition 1 based on the parameters used in practice, and explain how the constants required for new programs can be determined without evaluating the entire pool of scenarios?