- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes to use a coding LLM to optimize a reusable inference program over frozen recommendation models. The program jointly specifies feature construction, model composition, execution structure, and numerical parameters. It is selected using aggregate validation feedback and subsequently deployed without additional LLM calls.
Critical strengths
This paper proposes a flexible strategy for automatically constructing recommendation pipelines.
Critical weaknesses
- Insufficient justification of the problem formulation, and unclear presentation in abstract and introduction. The central distinction, query-level and system-level test-time scaling in recommendation, is not convincing. As far as I am concerned, conventional recommender systems already extensively optimize reusable ranking and inference pipelines offline. The paper does not sufficiently distinguish its proposed system-level TTS from these established practices. Furthermore, the abstract keeps talking about test-time scaling, and provides little explanation as to this problem in recommendation.
- Limited methodology innovation. The proposed approach uses a coding LLM to iteratively generate and refine programs based on aggregated validation scores. It does not introduce new algorithms or special designs to fit the challenges in the recommendation domain.
- Insufficient experiments. The benchmarks are extremely small.
Questions for authors
What is the so-called 'system-level test time scaling' in recommendation? Is it the auto-ML of recommendation pipeline on the arrangements of trained recommendation models and selection of features?
6 comments
W2: Methodological contribution and recommendation-specific design
The recommendation-specific design is specified in Secs. 3.1–3.2 of the submission. The methodological contribution concerns both the structure of the program space and the organization of feedback for searching it.
A structured recommendation-program space. Each program specifies feature construction F, an execution graph G, and numerical parameters Θ (Sec. 3.1, Eqs. (7)–(10)). Structural operators support candidate construction, conditional branching, and multi-stage retrieval and reranking; composition operators coordinate the frozen models through score or rank aggregation. The coding LLM constructs features and determines where they enter this computation. A feature can guide model composition or branch selection without being used directly as an item score. These choices address the coupled nature of recommendation inference: the value of a feature depends on how the available models and execution stages use it.
Two complementary feedback channels. ProgramRec organizes discovery around executability and recommendation-performance feedback (Sec. 3.2):
- Executability feedback reports contract violations and target-free runtime diagnostics, allowing the LLM to repair a candidate before formal ranking evaluation. This helps feature computation, model composition, and routing satisfy a common interface and execution contract (Eq. (11) and the accompanying text).
- Recommendation-performance feedback associates each submitted program's complete source with its aggregate validation score, its differences from the frozen reference and previous incumbent, and whether it becomes the new incumbent. This ordered history guides subsequent proposals (Eqs. (14)–(16)). Since ranking quality is observed for the complete program, without supervision for individual design decisions, retaining source code together with performance changes lets the LLM compare and revise the interacting choices of features, models, and execution structure.
The first channel helps construct executable recommendation programs; the second guides improvements in how their components work together. This is the recommendation-specific organization of the search described in the paper.
Evidence for the role of search history. We additionally compare ProgramRec with a no-history baseline using the same DeepSeek model and a budget of ten valid program evaluations. No-history retains the original discovered features, frozen recommenders, and within-candidate checking and repair, but independently generates candidates without previous program sources or validation outcomes. Both methods select their final program by validation performance. The table reports test NDCG@10, mean ± sample standard deviation over three trajectories.
| Dataset | No-history (B10) | ProgramRec (B10) |
|---|---|---|
| Amazon Movies & TV | 0.0207 ± 0.0012 | 0.0427 ± 0.0064 |
| Goodreads | 0.0697 ± 0.0121 | 0.1128 ± 0.0072 |
| Yelp | 0.0454 ± 0.0003 | 0.0484 ± 0.0014 |
ProgramRec achieves higher mean test performance on all three datasets at the same evaluated-program budget. This supports the contribution of accumulated program-and-outcome history beyond independent generation and validation-based selection alone.
W3: Experimental scale and additional baselines
We added experiments on a substantially larger Amazon Reviews 2023 Musical Instruments subset, containing 57,439 users, 24,587 items, and 396,958 training user–item interactions. Its user population and catalog are respectively 13.3× and 7.2× those of the largest original benchmark. Discovery uses aggregate validation feedback from 40,207 users, and the selected, frozen programs are evaluated on the complete test set of 57,439 users with full-catalog ranking.
We also expanded the original SREC, LightGCN, and SimpleX pool with HSTU and BERT4Rec. HSTU is the strongest individual model under this evaluation protocol, exceeding the original pool's strongest model, SimpleX. The table below reports the five individual recommenders and methods operating over the expanded five-model pool. ProgramRec and FunSearch both use DeepSeek here. Values with ± are means and sample standard deviations over three search trajectories.
| Method | Test NDCG@5 | Test NDCG@10 |
|---|---|---|
| SREC | 0.0186 | 0.0242 |
| LightGCN | 0.0192 | 0.0247 |
| SimpleX | 0.0194 | 0.0251 |
| HSTU | 0.0203 | 0.0259 |
| BERT4Rec | 0.0195 | 0.0245 |
| StaticFusion | 0.0250 | 0.0318 |
| RouteRec | 0.0185 | 0.0243 |
| RandomSearch | 0.0248 ± 0.0020 | 0.0306 ± 0.0025 |
| FunSearch–DeepSeek | 0.0238 ± 0.0029 | 0.0302 ± 0.0035 |
| ProgramRec–DeepSeek | 0.0273 ± 0.0004 | 0.0339 ± 0.0007 |
ProgramRec, FunSearch, and RandomSearch use B = 10 validation-scored programs. We did not include RecToolAgent in this larger-scale experiment because its per-request LLM calls over the full test set would exceed our available rebuttal resource budget.
ProgramRec–DeepSeek improves mean test NDCG@10 by 30.7% over HSTU, 6.3% over StaticFusion, and 12.2% over FunSearch–DeepSeek. Thus, substantial gains remain over the strongest individual recommender in the expanded pool, as well as over methods that already combine its models.
We additionally evaluated ProgramRec with three coding LLMs under two base-recommender configurations. The three-base pool contains SREC, LightGCN, and SimpleX; the five-base pool adds HSTU and BERT4Rec to these same three recommenders. These counts refer to the frozen recommenders available to the generated program; each row identifies the coding LLM used for discovery.
| Coding LLM | 3 base recommenders: NDCG@5 | 3 base recommenders: NDCG@10 | 5 base recommenders: NDCG@5 | 5 base recommenders: NDCG@10 |
|---|---|---|---|---|
| DeepSeek | 0.0247 ± 0.0024 | 0.0310 ± 0.0025 | 0.0273 ± 0.0004 | 0.0339 ± 0.0007 |
| GPT-Luna | 0.0218 ± 0.0021 | 0.0277 ± 0.0022 | 0.0267 ± 0.0021 | 0.0332 ± 0.0022 |
| Kimi | 0.0268 ± 0.0007 | 0.0332 ± 0.0009 | 0.0253 ± 0.0011 | 0.0316 ± 0.0011 |
All three variants exceed the strongest individual recommender in both pool configurations. These results show that the benefits of inference-program discovery persist with a substantially larger user population and catalog, and with an expanded pool containing a stronger base recommender.
References
[1] Lyu et al. Exploring Test-time Scaling via Prediction Merging on Large-Scale Recommendation. SIGIR 2026.
[2] Hutter, Kotthoff, and Vanschoren (eds.). Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019.
[3] Feurer et al. Efficient and Robust Automated Machine Learning. NeurIPS 2015.
Thank you for reviewing our paper and for raising questions about the problem formulation, methodological contribution, and experimental scale. We appreciate the opportunity to clarify the intended setting and its relationship to existing recommendation-pipeline optimization.
W1 and Q: System-level test-time scaling and its relationship to recommendation-pipeline optimization
We respectfully disagree with the assessment that the distinction between query-level and system-level scaling is not convincing. Our conceptual contribution is to identify and empirically study a new scaling axis for recommendation: allocating additional computation to discovering the reusable inference program over a fixed recommender stack. Recommendation scaling has primarily emphasized larger trained models; recent work extends this to additional predictions for each request and their merging. [1] We investigate a further dimension: improving the shared procedure that constructs features, coordinates available models, and produces rankings.
This distinction is explicitly defined in Sec. 2, Eqs. (4)–(6) of the submission. The recommendation stack, execution contract, validation evaluator, and coding LLM remain fixed as the discovery budget B increases. B counts contract-valid program evaluations. The selected program is then frozen and reused across unseen requests, while its execution can remain request-conditioned. Serving requires no further coding-LLM calls or recommendation-model updates.
We agree that reusable recommendation pipelines have long been optimized offline. The research question here is whether additional coding-LLM computation can systematically improve this inference layer, and whether the resulting gains transfer to unseen recommendation requests. Coding LLMs provide a means to construct, execute, and revise complete programs, making this layer a practical object of computational search. Existing offline optimization establishes the relevance of the object; our formulation and experiments examine its behavior as discovery budget increases.
Relationship to AutoML. AutoML broadly automates design decisions in machine-learning workflows, including feature processing, model selection, hyperparameters, and pipeline composition. [2] A classical example, Auto-sklearn, selects algorithms, preprocessing, and hyperparameters to build predictive models for a dataset. [3] ProgramRec starts from already-trained recommenders and searches their inference program while keeping their weights fixed. Its search includes feature construction, conditional execution structure, model composition, and numerical parameters (Sec. 3.1). There is a broad connection to AutoML, but that categorization does not erase the specific contribution: defining and evaluating how discovery compute improves a shared recommendation program under a fixed model stack.
Evidence for this scaling dimension. In addition to the original budget-checkpoint experiments (Appendix A.3.1), we extended one original Yelp–DeepSeek discovery trajectory from B10 to B50. At each checkpoint, the program is selected by validation performance and subsequently evaluated on test; test results are not used to guide discovery.
Along this trajectory, increasing discovery budget from B10 to B50 improves test NDCG@10 by 10.26%, with the base recommenders unchanged. This provides direct evidence that additional computation invested in inference-program discovery can yield further recommendation gains beyond the original budget.
Presentation. The submitted abstract already describes reusable programs over frozen recommendation models (lines 11–24). We will make the distinction between request-level execution compute and pre-deployment program-discovery compute more explicit in the abstract and introduction.