acceptodds
Under review as a conference paper at ICLR 2027

Scaling the Inference Program: System-Level Test-Time Scaling for Recommendation

17 views

Abstract

Test-time scaling improves predictions without updating model weights, but it typically spends additional computation independently for each query. Recommendation exposes a distinct system-level opportunity: various requests repeatedly pass through the same frozen recommendation models, making the reusable, request-conditioned inference program a natural object on which to spend additional compute. We introduce System-Level Test-Time Scaling and instantiate it with the proposed framework ProgramRec. Starting from a blank program interface, a large language model iteratively constructs contract-valid inference programs that jointly specify feature construction , a recursive execution graph , and numerical parameters . Aggregate validation feedback and the history of evaluated programs guide subsequent proposals. The selected program is then frozen and reused deterministically across requests without further LLM calls or model updates. Across three datasets, the strongest ProgramRec variant achieves the best overall performance, while larger discovery budgets yield gains in every setting. These results establish reusable inference-program construction as a new scaling axis for recommendation. The code is available at https://anonymous.4open.science/r/ProgramRec-0675.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Scaling, Recommendation Agent, System-Level

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_EjntOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes to use a coding LLM to optimize a reusable inference program over frozen recommendation models. The program jointly specifies feature construction, model composition, execution structure, and numerical parameters. It is selected using aggregate validation feedback and subsequently deployed without additional LLM calls.

Critical strengths

This paper proposes a flexible strategy for automatically constructing recommendation pipelines.

Critical weaknesses
  1. Insufficient justification of the problem formulation, and unclear presentation in abstract and introduction. The central distinction, query-level and system-level test-time scaling in recommendation, is not convincing. As far as I am concerned, conventional recommender systems already extensively optimize reusable ranking and inference pipelines offline. The paper does not sufficiently distinguish its proposed system-level TTS from these established practices. Furthermore, the abstract keeps talking about test-time scaling, and provides little explanation as to this problem in recommendation.
  2. Limited methodology innovation. The proposed approach uses a coding LLM to iteratively generate and refine programs based on aggregated validation scores. It does not introduce new algorithms or special designs to fit the challenges in the recommendation domain.
  3. Insufficient experiments. The benchmarks are extremely small.
Questions for authors

What is the so-called 'system-level test time scaling' in recommendation? Is it the auto-ML of recommendation pipeline on the arrangements of trained recommendation models and selection of features?

6 comments

Thank you for reviewing our paper and for raising questions about the problem formulation, methodological contribution, and experimental scale. We appreciate the opportunity to clarify the intended setting and its relationship to existing recommendation-pipeline optimization.

W1 and Q: System-level test-time scaling and its relationship to recommendation-pipeline optimization

We respectfully disagree with the assessment that the distinction between query-level and system-level scaling is not convincing. Our conceptual contribution is to identify and empirically study a new scaling axis for recommendation: allocating additional computation to discovering the reusable inference program over a fixed recommender stack. Recommendation scaling has primarily emphasized larger trained models; recent work extends this to additional predictions for each request and their merging. [1] We investigate a further dimension: improving the shared procedure that constructs features, coordinates available models, and produces rankings.

Scaling settingWhere additional computation is allocatedWhat it improves
Training-time scalingTraining the recommendation modelsLearned model parameters
Query-level test-time scalingExecuting additional computation for each incoming requestThe prediction produced for that request
System-level scaling studied hereDiscovering an inference program after model training and before deploymentThe shared recommendation procedure, with base-model weights fixed

This distinction is explicitly defined in Sec. 2, Eqs. (4)–(6) of the submission. The recommendation stack, execution contract, validation evaluator, and coding LLM remain fixed as the discovery budget B increases. B counts contract-valid program evaluations. The selected program is then frozen and reused across unseen requests, while its execution can remain request-conditioned. Serving requires no further coding-LLM calls or recommendation-model updates.

We agree that reusable recommendation pipelines have long been optimized offline. The research question here is whether additional coding-LLM computation can systematically improve this inference layer, and whether the resulting gains transfer to unseen recommendation requests. Coding LLMs provide a means to construct, execute, and revise complete programs, making this layer a practical object of computational search. Existing offline optimization establishes the relevance of the object; our formulation and experiments examine its behavior as discovery budget increases.

Relationship to AutoML. AutoML broadly automates design decisions in machine-learning workflows, including feature processing, model selection, hyperparameters, and pipeline composition. [2] A classical example, Auto-sklearn, selects algorithms, preprocessing, and hyperparameters to build predictive models for a dataset. [3] ProgramRec starts from already-trained recommenders and searches their inference program while keeping their weights fixed. Its search includes feature construction, conditional execution structure, model composition, and numerical parameters (Sec. 3.1). There is a broad connection to AutoML, but that categorization does not erase the specific contribution: defining and evaluating how discovery compute improves a shared recommendation program under a fixed model stack.

Evidence for this scaling dimension. In addition to the original budget-checkpoint experiments (Appendix A.3.1), we extended one original Yelp–DeepSeek discovery trajectory from B10 to B50. At each checkpoint, the program is selected by validation performance and subsequently evaluated on test; test results are not used to guide discovery.

Discovery budget BValidation NDCG@10Test NDCG@10
100.0559650.049673
200.0567610.050747
300.0569700.051175
500.0581830.054767

Along this trajectory, increasing discovery budget from B10 to B50 improves test NDCG@10 by 10.26%, with the base recommenders unchanged. This provides direct evidence that additional computation invested in inference-program discovery can yield further recommendation gains beyond the original budget.

Presentation. The submitted abstract already describes reusable programs over frozen recommendation models (lines 11–24). We will make the distinction between request-level execution compute and pre-deployment program-discovery compute more explicit in the abstract and introduction.

W2: Methodological contribution and recommendation-specific design

The recommendation-specific design is specified in Secs. 3.1–3.2 of the submission. The methodological contribution concerns both the structure of the program space and the organization of feedback for searching it.

A structured recommendation-program space. Each program specifies feature construction F, an execution graph G, and numerical parameters Θ (Sec. 3.1, Eqs. (7)–(10)). Structural operators support candidate construction, conditional branching, and multi-stage retrieval and reranking; composition operators coordinate the frozen models through score or rank aggregation. The coding LLM constructs features and determines where they enter this computation. A feature can guide model composition or branch selection without being used directly as an item score. These choices address the coupled nature of recommendation inference: the value of a feature depends on how the available models and execution stages use it.

Two complementary feedback channels. ProgramRec organizes discovery around executability and recommendation-performance feedback (Sec. 3.2):

  • Executability feedback reports contract violations and target-free runtime diagnostics, allowing the LLM to repair a candidate before formal ranking evaluation. This helps feature computation, model composition, and routing satisfy a common interface and execution contract (Eq. (11) and the accompanying text).
  • Recommendation-performance feedback associates each submitted program's complete source with its aggregate validation score, its differences from the frozen reference and previous incumbent, and whether it becomes the new incumbent. This ordered history guides subsequent proposals (Eqs. (14)–(16)). Since ranking quality is observed for the complete program, without supervision for individual design decisions, retaining source code together with performance changes lets the LLM compare and revise the interacting choices of features, models, and execution structure.

The first channel helps construct executable recommendation programs; the second guides improvements in how their components work together. This is the recommendation-specific organization of the search described in the paper.

Evidence for the role of search history. We additionally compare ProgramRec with a no-history baseline using the same DeepSeek model and a budget of ten valid program evaluations. No-history retains the original discovered features, frozen recommenders, and within-candidate checking and repair, but independently generates candidates without previous program sources or validation outcomes. Both methods select their final program by validation performance. The table reports test NDCG@10, mean ± sample standard deviation over three trajectories.

DatasetNo-history (B10)ProgramRec (B10)
Amazon Movies & TV0.0207 ± 0.00120.0427 ± 0.0064
Goodreads0.0697 ± 0.01210.1128 ± 0.0072
Yelp0.0454 ± 0.00030.0484 ± 0.0014

ProgramRec achieves higher mean test performance on all three datasets at the same evaluated-program budget. This supports the contribution of accumulated program-and-outcome history beyond independent generation and validation-based selection alone.

W3: Experimental scale and additional baselines

We added experiments on a substantially larger Amazon Reviews 2023 Musical Instruments subset, containing 57,439 users, 24,587 items, and 396,958 training user–item interactions. Its user population and catalog are respectively 13.3× and 7.2× those of the largest original benchmark. Discovery uses aggregate validation feedback from 40,207 users, and the selected, frozen programs are evaluated on the complete test set of 57,439 users with full-catalog ranking.

We also expanded the original SREC, LightGCN, and SimpleX pool with HSTU and BERT4Rec. HSTU is the strongest individual model under this evaluation protocol, exceeding the original pool's strongest model, SimpleX. The table below reports the five individual recommenders and methods operating over the expanded five-model pool. ProgramRec and FunSearch both use DeepSeek here. Values with ± are means and sample standard deviations over three search trajectories.

MethodTest NDCG@5Test NDCG@10
SREC0.01860.0242
LightGCN0.01920.0247
SimpleX0.01940.0251
HSTU0.02030.0259
BERT4Rec0.01950.0245
StaticFusion0.02500.0318
RouteRec0.01850.0243
RandomSearch0.0248 ± 0.00200.0306 ± 0.0025
FunSearch–DeepSeek0.0238 ± 0.00290.0302 ± 0.0035
ProgramRec–DeepSeek0.0273 ± 0.00040.0339 ± 0.0007

ProgramRec, FunSearch, and RandomSearch use B = 10 validation-scored programs. We did not include RecToolAgent in this larger-scale experiment because its per-request LLM calls over the full test set would exceed our available rebuttal resource budget.

ProgramRec–DeepSeek improves mean test NDCG@10 by 30.7% over HSTU, 6.3% over StaticFusion, and 12.2% over FunSearch–DeepSeek. Thus, substantial gains remain over the strongest individual recommender in the expanded pool, as well as over methods that already combine its models.

We additionally evaluated ProgramRec with three coding LLMs under two base-recommender configurations. The three-base pool contains SREC, LightGCN, and SimpleX; the five-base pool adds HSTU and BERT4Rec to these same three recommenders. These counts refer to the frozen recommenders available to the generated program; each row identifies the coding LLM used for discovery.

Coding LLM3 base recommenders: NDCG@53 base recommenders: NDCG@105 base recommenders: NDCG@55 base recommenders: NDCG@10
DeepSeek0.0247 ± 0.00240.0310 ± 0.00250.0273 ± 0.00040.0339 ± 0.0007
GPT-Luna0.0218 ± 0.00210.0277 ± 0.00220.0267 ± 0.00210.0332 ± 0.0022
Kimi0.0268 ± 0.00070.0332 ± 0.00090.0253 ± 0.00110.0316 ± 0.0011

All three variants exceed the strongest individual recommender in both pool configurations. These results show that the benefits of inference-program discovery persist with a substantially larger user population and catalog, and with an expanded pool containing a stronger base recommender.

References

[1] Lyu et al. Exploring Test-time Scaling via Prediction Merging on Large-Scale Recommendation. SIGIR 2026.

[2] Hutter, Kotthoff, and Vanschoren (eds.). Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019.

[3] Feurer et al. Efficient and Robust Automated Machine Learning. NeurIPS 2015.

Reviewer_ZzhGOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether additional computation can improve a reusable recommendation program rather than requiring an LLM for every recommendation request. The proposed framework, ProgramRec, uses a coding LLM to generate candidate programs from earlier programs and aggregate validation feedback. Each candidate determines how features and frozen recommender outputs are used within a bounded graph of retrieval, routing, composition, and reranking operations. After search, the program with the best validation score is frozen and reused without further coding-LLM calls or updates to the underlying recommenders. The authors call this system-level test-time scaling and report improved recommendation performance, gains with larger discovery budgets, and transfer of programs across datasets. Additional analyses examine the components of selected programs and the role of search history.

Critical strengths
  • Separating discovery from deployment has clear practical value. The LLM is used during search, while deployment executes a fixed, bounded program over frozen recommenders. The paper explicitly separates validation selection from test evaluation and describes program freezing, deterministic checks, and restricted feedback access.

  • The experiments cover different datasets and coding LLMs and report variation across independent search trajectories. Gains over the listed baselines are substantial on some datasets, and cross-dataset program transfer provides useful evidence of reuse.

Critical weaknesses

The comparisons do not fully separate the search strategy from the richer feature interface. StaticFusion combines expert outputs, while context evidence has a large effect in the ablation. A simple reranker using the same evidence features is therefore an important missing control. The no-history comparison covers only one trajectory, and component ablations modify selected programs on validation without reoptimization. These results support the selected programs but do not yet establish the general advantage of the proposed discovery strategy over simpler alternatives.

Questions for authors

How does ProgramRec compare with simple reranking or fixed-structure parameter search using the same features and matched evaluation budgets? Does the benefit of search history hold across datasets and independent runs?

3 comments

Thank you for recognizing the practical value of separating program discovery from deployment and the evidence of program reuse. We appreciate your suggestions to distinguish the contribution of the search strategy from the feature interface and to strengthen the analysis of search history.

Simple reranking and fixed-structure parameter search

We added two simpler controls. Simple reranking merges the frozen recommenders' Top-10 lists, removes duplicates by keeping each item's highest raw score, and returns the overall Top-10. It has no learned parameters or validation-based tuning. Fixed-structure parameter search uses Bayesian optimization (BO) to optimize the global mixture weights of normalized expert scores while keeping the fusion structure fixed. The optimizer receives only aggregate validation NDCG@10, with B = 10 evaluations, matching ProgramRec's evaluated-program budget.

The table reports full-test NDCG@5 and NDCG@10. BO and ProgramRec use three search trajectories, reported as mean ± sample standard deviation; the deterministic reranker is evaluated once. ProgramRec uses DeepSeek and retains the original main-table results.

DatasetMethodNDCG@5NDCG@10
Amazon Movies & TVSimple reranker0.01000.0146
Amazon Movies & TVFixed-structure BO0.0187 ± 0.00030.0254 ± 0.0008
Amazon Movies & TVProgramRec0.0369 ± 0.00570.0427 ± 0.0064
GoodreadsSimple reranker0.02790.0374
GoodreadsFixed-structure BO0.0448 ± 0.00100.0588 ± 0.0006
GoodreadsProgramRec0.0987 ± 0.00690.1128 ± 0.0072
YelpSimple reranker0.03090.0379
YelpFixed-structure BO0.0382 ± 0.00000.0464 ± 0.0000
YelpProgramRec0.0395 ± 0.00170.0484 ± 0.0014

ProgramRec's mean advantage is larger on Amazon and Goodreads. On Yelp, fixed-structure BO is already competitive: its test NDCG@10 is 0.0464, compared with 0.0484 for ProgramRec, a relative improvement of 4.3%. Thus, simple model fusion is a strong alternative on Yelp, and the additional benefit of ProgramRec at B10 is modest. These results indicate that the benefit of more flexible program discovery varies across datasets. We additionally examine search with the discovered features retained below.

No-history comparison across datasets and independent trajectories

We expanded the no-history comparison to all three original datasets, with three independent search trajectories per dataset. Both methods use DeepSeek, the same ProgramRec-discovered features, frozen recommenders, and B = 10 evaluated programs. No-history removes access to previous candidate sources and validation outcomes when constructing the next candidate, while retaining execution checks and repair within the current candidate. Both methods select the final program by validation NDCG@10 and then evaluate the frozen program on test.

DatasetNo-history NDCG@5ProgramRec NDCG@5No-history NDCG@10ProgramRec NDCG@10
Amazon Movies & TV0.0154 ± 0.00150.0369 ± 0.00570.0207 ± 0.00120.0427 ± 0.0064
Goodreads0.0543 ± 0.01100.0987 ± 0.00690.0697 ± 0.01210.1128 ± 0.0072
Yelp0.0376 ± 0.00120.0395 ± 0.00170.0454 ± 0.00030.0484 ± 0.0014

All entries are test metrics, reported as mean ± sample standard deviation. The higher ProgramRec means across all three datasets support the contribution of accumulated program-and-performance history beyond independent generation and validation-based selection alone. This evidence extends the original single-trajectory comparison while retaining the observed search variation.

Component ablations and comparison with simpler search alternatives

The component ablations in Sec. 4.4 and Appendix A.3.2 measure how particular components contribute within the selected, frozen programs. Because these transformations do not include rediscovery, they do not establish the best performance attainable after removing a feature or operator from the search space. The additional search comparisons above provide separate evidence about the discovery procedure.

We also added a conditional RandomSearch control using the DeepSeek-discovered features on the larger Amazon Reviews 2023 Musical Instruments subset, with 57,439 test users and 24,587 catalog items. It searches structures from the same graph-operator vocabulary without using earlier performance feedback to guide proposals. The five-model pool contains SREC, LightGCN, SimpleX, HSTU, and BERT4Rec.

MethodTest NDCG@5Test NDCG@10
Simple reranker0.02000.0256
Fixed-structure BO0.0247 ± 0.00050.0311 ± 0.0005
RandomSearch with DeepSeek-discovered features0.0248 ± 0.00200.0306 ± 0.0025
ProgramRec–DeepSeek0.0273 ± 0.00040.0339 ± 0.0007

The search methods use B = 10 validation evaluations and report three-trajectory means ± sample standard deviations; RandomSearch's budget includes its initial reference program. ProgramRec achieves higher mean test performance than these alternatives. Together with the no-history comparison, these results provide direct evidence for the value of guided discovery beyond the local component-ablation analysis.

Dear Reviewer,

We're following up on our response. As the discussion period is nearing its end, we'd be grateful for an update on whether it addresses your concerns, or if any further clarification is needed.

Thank you for your time.

Best regards,

The Authors

Reviewer_bkZrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose ProgramRec, where a coding LLM repeatedly writes and refines code that combines frozen recommender models, scoring each version on validation data, and the best program is then frozen and deployed with zero LLM calls. It beats all baselines on Amazon, Goodreads, and Yelp, improving NDCG@10 by up to 74.6% over the strongest one.

Critical strengths
  1. The final program is plain frozen code with no LLM calls at serving time, so it’s cheap and fast enough to actually deploy.
  2. With just B = 10, it beats the selected baseline on all three datasets.
Critical weaknesses
  1. Limited novelty and a very domain-specific setting: the method is essentially FunSearch/AlphaEvolve-style LLM program search applied to recommendation, and no such method is included as a baseline. (missing baselines)

  2. The datasets are tiny, and validation is reused. There are only about 1k–4k users, with one validation item per user, and the LLM sees the validation score after every try. With that little data, the search can overfit validation. The small Yelp gain (5.8%) and the wide variance bands are an indication.

  3. Weak base models: absolute scores are low.

Questions for authors
  1. How much of the gain is left if B grows?

  2. How does ProgramRec compare with existing LLM program-search methods like FunSearch or AlphaEvolve, given the same program space and budget?

  3. Do the gains still hold with stronger base recommenders or larger datasets, where single models are already much better?

4 comments

Thank you for recognizing the practical value of a frozen inference program that requires no LLM calls at serving time. We appreciate your suggestions to clarify the relationship to existing program-search methods and to strengthen the evaluation with larger datasets and stronger base recommenders.

W1 and Q2: Novelty and comparison with existing LLM program-search methods

Contribution and relevance to recommendation. Our contribution has two parts: identifying the reusable inference program as a meaningful scaling object for recommendation, and developing a search method for this structured program space (Sections 2–3). Representative LLM-based rankers spend LLM computation on producing rankings for individual requests. [1] We investigate a complementary question: can additional LLM computation improve the inference procedure shared across future requests, while keeping the underlying recommenders frozen? The resulting program makes request-dependent decisions but requires no further LLM calls or model updates. This offers another dimension for improving recommendation beyond making the LLM itself a better recommender.

This optimization object is practically important. Industrial recommendation already relies on executable logic that coordinates candidate sources, filtering, ranking, and diversification. For example, Instagram's production recommendation system describes these operations through a recommendation-specific language. [2] Their composition determines how the available models are used. ProgramRec studies whether LLM discovery compute can systematically improve this layer, making inference-program construction itself an object of recommendation optimization.

Search design and relationship to FunSearch. ProgramRec belongs to the broader family of LLM program-search methods, including FunSearch. [3] Within this family, the organization of context and feedback is an important design choice. Recent Context Language Models also investigate open-ended discovery with agent-managed context, including comparisons with specialized evolutionary workflows. [4]

RecRSI, first posted to arXiv in November 2026, after our submission, also studies recommendation-specific algorithm discovery, using mechanism hypotheses and accumulated experimental evidence to guide changes to recommender algorithms. Its search trains and evaluates the modified algorithms. [5] ProgramRec addresses a complementary optimization target: the reusable inference program coordinating already-trained, frozen recommenders. Here, features, models, and execution stages must work together. In Section 3.1, a complete program specifies feature construction, an execution graph, and numerical parameters. Features can control model composition or conditional branches; they are not restricted to additional inputs to a fixed final scoring function.

ProgramRec organizes this search around two complementary forms of feedback (Section 3.2). Executability feedback supplies contract violations and target-free runtime diagnostics, allowing the LLM to revise a candidate before submission. Recommendation-performance feedback supplies the complete program's aggregate validation result, together with its relation to the reference model and previous incumbent, through an explicit program-and-outcome history. FunSearch also uses evaluated programs and history: its evolutionary procedure selects successful program examples from a scored population to guide generation. [3]

This feedback organization addresses two difficulties of recommendation-program search. First, feature construction, model composition, and routing must satisfy a common execution contract; diagnostic feedback lets the LLM repair incompatible interfaces or numerical failures before ranking evaluation. Second, these decisions interact: a feature can be useful for choosing a model or branch even when it is not useful as a direct item score. Yet recommendation quality is observed only through the complete program's aggregate ranking metric, without supervision for individual design decisions. Keeping source code together with explicit performance changes allows the LLM to compare successive designs and revise these coupled choices. The two feedback channels therefore support both constructing a valid recommendation pipeline and improving how its frozen models work together.

Additional FunSearch comparison. We added the official FunSearch implementation over the same frozen recommendation stack and validation objective. FunSearch can construct its own features and search the composition of the base models. Below we compare FunSearch–DeepSeek with the original ProgramRec–DeepSeek results, at B = 10 validation-scored programs. Results are test NDCG, reported as mean ± standard deviation over three search trajectories; programs are selected using validation performance.

DatasetMethodNDCG@5NDCG@10
Amazon Movies & TVFunSearch0.0249 ± 0.01240.0311 ± 0.0124
Amazon Movies & TVProgramRec0.0369 ± 0.00570.0427 ± 0.0064
GoodreadsFunSearch0.0474 ± 0.00410.0617 ± 0.0030
GoodreadsProgramRec0.0987 ± 0.00690.1128 ± 0.0072
YelpFunSearch0.0354 ± 0.00180.0442 ± 0.0016
YelpProgramRec0.0395 ± 0.00170.0484 ± 0.0014

This comparison addresses the missing program-search baseline and shows that ProgramRec is competitive at the evaluated budget. It complements our central contributions: the recommendation scaling formulation and the search design for its structured inference programs.

W2 (dataset scale), W3 and Q3: Larger datasets and stronger base recommenders

We added experiments on a substantially larger Amazon Reviews 2023 Musical Instruments subset, containing 57,439 users, 24,587 items, and 396,958 training user–item interactions. Its user population and catalog are respectively 13.3× and 7.2× those of the largest original benchmark. Discovery uses aggregate validation feedback from 40,207 users, and the selected, frozen programs are evaluated on the complete test set of 57,439 users with full-catalog ranking.

We also expanded the original SREC, LightGCN, and SimpleX pool with HSTU and BERT4Rec. HSTU is the strongest individual model under this evaluation protocol, exceeding the original pool's strongest model, SimpleX. The table below reports the five individual recommenders and methods operating over the expanded five-model pool. ProgramRec and FunSearch both use DeepSeek here. Values with ± are means and standard deviations over three search trajectories.

MethodTest NDCG@5Test NDCG@10
SREC0.01860.0242
LightGCN0.01920.0247
SimpleX0.01940.0251
HSTU0.02030.0259
BERT4Rec0.01950.0245
StaticFusion0.02500.0318
RouteRec0.01850.0243
RandomSearch0.0248 ± 0.00200.0306 ± 0.0025
FunSearch–DeepSeek0.0238 ± 0.00290.0302 ± 0.0035
ProgramRec–DeepSeek0.0273 ± 0.00040.0339 ± 0.0007

ProgramRec, FunSearch, and RandomSearch use B = 10 validation-scored programs. We did not include RecToolAgent in this larger-scale experiment because its per-request LLM calls over the full test set would exceed our available rebuttal resource budget.

ProgramRec–DeepSeek improves mean test NDCG@10 by 30.7% over HSTU, 6.3% over StaticFusion, and 12.2% over FunSearch–DeepSeek. Thus, substantial gains remain over the strongest individual recommender in the expanded pool, as well as over methods that already combine its models.

We additionally evaluated ProgramRec with three coding LLMs under two base-recommender configurations. The three-base pool contains SREC, LightGCN, and SimpleX; the five-base pool adds HSTU and BERT4Rec to these same three recommenders. These counts refer to the frozen recommenders available to the generated program; each row identifies the coding LLM used for discovery.

Coding LLM3 base recommenders: NDCG@53 base recommenders: NDCG@105 base recommenders: NDCG@55 base recommenders: NDCG@10
DeepSeek0.0247 ± 0.00240.0310 ± 0.00250.0273 ± 0.00040.0339 ± 0.0007
GPT-Luna0.0218 ± 0.00210.0277 ± 0.00220.0267 ± 0.00210.0332 ± 0.0022
Kimi0.0268 ± 0.00070.0332 ± 0.00090.0253 ± 0.00110.0316 ± 0.0011

All three variants exceed the strongest individual recommender in both pool configurations. These results show that the benefits of inference-program discovery persist with a substantially larger user population and catalog, and with an expanded pool containing a stronger base recommender.

W2 (validation reuse and search variation) and Q1: Generalization and gains at larger budgets

Validation reuse and transfer to test. Our evidence for generalization comes from the test performance of the validation-selected programs. In the original experiments, all candidates within a dataset's search are evaluated on the same validation split (Sec. 2, Eq. (5)). Candidate source code and aggregate validation feedback are appended to the discovery history and used to guide subsequent proposals (Sec. 3.2, Eqs. (11)–(16)). Per-request validation targets, ranks, and metric contributions are not exposed to the coding LLM. At each budget, the validation winner is selected and frozen before test evaluation; test results are never used for proposal generation or program selection (Appendix A.2.3, p. 22).

We compare validation and test performance of the same selected program at the first valid candidate (B1) and the final recorded checkpoint. All entries are NDCG@10, reported as mean ± sample standard deviation over three trajectories.

DatasetCoding LLMValidation B1Validation finalTest B1Test final
Amazon Movies & TVDeepSeek0.0362 ± 0.00680.0690 ± 0.00600.0203 ± 0.00620.0427 ± 0.0064
Amazon Movies & TVGPT0.0380 ± 0.00090.0586 ± 0.00740.0228 ± 0.00100.0373 ± 0.0054
Amazon Movies & TVKimi0.0374 ± 0.00390.0593 ± 0.00600.0209 ± 0.00260.0316 ± 0.0067
GoodreadsDeepSeek0.0687 ± 0.01250.1299 ± 0.00500.0537 ± 0.00840.1128 ± 0.0072
GoodreadsGPT0.0775 ± 0.01320.1066 ± 0.02120.0617 ± 0.00930.0894 ± 0.0205
GoodreadsKimi0.0680 ± 0.00690.1229 ± 0.01020.0528 ± 0.00480.1035 ± 0.0164
YelpDeepSeek0.0468 ± 0.00450.0539 ± 0.00210.0426 ± 0.00210.0484 ± 0.0014
YelpGPT0.0475 ± 0.00370.0497 ± 0.00750.0421 ± 0.00160.0435 ± 0.0042
YelpKimi0.0456 ± 0.00400.0547 ± 0.00240.0409 ± 0.00520.0489 ± 0.0019

Final denotes B10, except for one Amazon–Kimi trajectory whose original endpoint is B9; that result is retained unchanged. Mean test performance improves in all nine dataset–LLM settings, including all three Yelp settings. The standard deviations retain the observed variation across search trajectories.

Yelp and additional discovery budget. Simple model combination already performs strongly on Yelp, which helps explain the smaller gain over the baselines: StaticFusion reaches test NDCG@10 of 0.0462, close to the best ProgramRec result of 0.0489 (Table 1). The component analysis provides a consistent structural explanation. In the reference DeepSeek program, removing multi-stage structure reduces validation NDCG@10 by only 0.0002 on Yelp, compared with 0.0028 on Amazon and 0.0099 on Goodreads (Sec. 4.4, Table 2). These observations suggest that relatively simple model composition captures much of the available benefit on Yelp, with less marginal benefit from elaborate multi-stage execution.

The smaller contribution of multi-stage structure does not imply that the discovered model composition is already fully optimized. Additional discovery budget allows further exploration of feature use, model combinations, and numerical parameters within the same execution contract. To test whether this search can yield further gains despite the strong simple-fusion baseline, we extended an original Yelp–DeepSeek trajectory from B10 to B50. At each checkpoint, the program is selected by validation performance and then evaluated on test:

Discovery budget BValidation NDCG@10Test NDCG@10
100.0559650.049673
200.0567610.050747
300.0569700.051175
500.0581830.054767

Increasing B from 10 to 50 improves validation NDCG@10 from 0.055965 to 0.058183, while test NDCG@10 increases from 0.049673 to 0.054767 (+10.26%). Thus, even on Yelp, where simple model fusion is already competitive, further discovery finds a better inference program whose validation gains transfer to test. The smaller gain at the original budget therefore does not imply that the search has exhausted the available improvements. This B50 continuation is a single-trajectory result; the preceding table separately reports variation over three trajectories per setting. Together, these observations support generalization of the measured gains without assuming that aggregate feedback eliminates adaptive selection bias.

References

[1] Yupeng Hou et al. Large Language Models are Zero-Shot Rankers for Recommender Systems. ECIR, 2024.

[2] Engineering at Meta. How Instagram suggests new content. 2020.

[3] Bernardino Romera-Paredes et al. Mathematical discoveries from program search with large language models. Nature, 625:468–475, 2024.

[4] Rulin Shao et al. Context Language Models. 2026.

[5] Tingran Gan et al. RecRSI: Mechanism-Guided Discovery of Recommendation Algorithms. 2026.

Dear Reviewer,

We're following up on our response. As the discussion period is nearing its end, we'd be grateful for an update on whether it addresses your concerns, or if any further clarification is needed.

Thank you for your time.

Best regards, The Authors

Reviewer_33DsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that conventional LLM inference treats requests as independent: additional computation benefits only the query at hand. In recommender systems, however, a single frozen model stack serves a massive volume of homogeneous requests. Its inference strategy can therefore be optimized offline and reused online, amortizing the cost of optimization rather than paying it anew for every request. Building on this insight, the authors propose PROGRAMREC, which shifts the focus of test-time scaling from query-level execution trajectories to reusable inference programs. Experiments across three recommendation datasets, three coding LLMs, and three frozen recommenders show that PROGRAMREC outperforms individual models, StaticFusion, RouteRec, RecToolAgent, and budget-matched random search on every dataset. Its performance also improves as the discovery budget grows.

Critical strengths
  1. The paper argues that requests in conventional LLM inference are isolated, so additional compute benefits only the query at hand. In contrast, a recommender serves massive numbers of homogeneous requests using the same frozen model stack, making its inference strategy an asset that can be optimized offline and amortized across requests.

  2. This paper applies a coding LLM repeatedly generate and evaluates inference programs before deployment, then freezes the best-performing program for subsequent requests. This avoids calling an LLM for every request at serving time, and the method improves performance on all three datasets.

  3. This paper reports results across three datasets, three coding LLMs, and three frozen recommenders, and compares against Random Search using the same program space and budget. Together, these results provide strong empirical support for the paper’s claims.

Critical weaknesses
  1. Repeated use of the validation set may introduce selection bias. Candidate scores are fed back to the LLM, and the final program is selected on the same set; aggregate feedback limits per-example leakage but may still lead the search to adapt to validation data. The paper does not quantify this effect, and with only three trajectories per setting, the robustness of gains from larger budgets remains unclear.
  2. The role of search feedback and the discovery cost could be analyzed further. The comparison with Random Search provides useful evidence for the method; adding a one-shot baseline with the same LLM or another search strategy would help clarify the contribution of cross-round feedback. In addition, budget B counts formal program evaluations, while candidate construction also involves multiple LLM calls; reporting results against actual discovery cost would make the scaling analysis more complete.
Questions for authors
  1. Could you clarify whether all candidate programs are evaluated on the same validation set, and how the accumulated validation feedback is used across discovery rounds? Any analysis of how this repeated evaluation affects the selected program’s generalization would help me interpret the reported gains from larger budgets.
  2. If available, could you compare the trajectory-guided approach with one-shot generation by the same LLM, or report their performance at a similar discovery cost? This would help clarify how much the iterative feedback contributes to the results.
3 comments

Thank you for the careful review and for recognizing the value of optimizing a reusable inference program over a frozen recommender stack. We appreciate your suggestions on validation reuse, the contribution of search feedback, and the accounting of discovery cost.

W1 and Q1: Validation reuse and generalization of gains at larger budgets

Yes. All submitted, contract-valid candidate programs for a dataset are evaluated once on the same complete validation split, with the validation environment and ranking evaluator fixed throughout discovery (Sec. 2, Eq. (5); Appendix A.2.3, p. 22). After each evaluation, we append the candidate's complete source and four aggregate feedback quantities to the ordered discovery history: its validation NDCG@10, its difference from the strongest frozen recommender, its difference from the previous best program, and whether it becomes the new best program (Sec. 3.2, Eqs. (14)–(16)). The coding LLM receives this accumulated history when constructing the next program's feature computation, controller structure, and numerical parameters (Sec. 3.2, Eq. (11)). Thus, validation feedback guides subsequent program construction as well as selection. Per-request targets, ranks, metric contributions, and branch counterfactuals are never exposed to the coding LLM (Appendix A.2.3, “Validation feedback and selection,” p. 22). At each budget, the program with the highest validation NDCG@10 is selected and frozen before test evaluation (Appendix A.3.1, pp. 22–23). Test outcomes are never added to the discovery history or used for proposal generation or program selection (Appendix A.2.3, “Frozen test execution,” p. 22).

To examine whether the gains from this repeated validation-guided search generalize, we compare the validation and test performance of the same selected program at each recorded budget checkpoint. Across the three datasets and three coding LLMs, the mean test NDCG@10 at the final recorded budget exceeds that of the first valid program in all nine settings. These comparisons retain the three trajectories per setting and their reported variation.

The table below summarizes the first valid program (B1) and the final recorded checkpoint. All entries are NDCG@10, reported as mean ± sample standard deviation over three trajectories.

DatasetCoding LLMValidation B1Validation finalTest B1Test final
Amazon Movies & TVDeepSeek0.0362 ± 0.00680.0690 ± 0.00600.0203 ± 0.00620.0427 ± 0.0064
Amazon Movies & TVGPT0.0380 ± 0.00090.0586 ± 0.00740.0228 ± 0.00100.0373 ± 0.0054
Amazon Movies & TVKimi0.0374 ± 0.00390.0593 ± 0.00600.0209 ± 0.00260.0316 ± 0.0067
GoodreadsDeepSeek0.0687 ± 0.01250.1299 ± 0.00500.0537 ± 0.00840.1128 ± 0.0072
GoodreadsGPT0.0775 ± 0.01320.1066 ± 0.02120.0617 ± 0.00930.0894 ± 0.0205
GoodreadsKimi0.0680 ± 0.00690.1229 ± 0.01020.0528 ± 0.00480.1035 ± 0.0164
YelpDeepSeek0.0468 ± 0.00450.0539 ± 0.00210.0426 ± 0.00210.0484 ± 0.0014
YelpGPT0.0475 ± 0.00370.0497 ± 0.00750.0421 ± 0.00160.0435 ± 0.0042
YelpKimi0.0456 ± 0.00400.0547 ± 0.00240.0409 ± 0.00520.0489 ± 0.0019

Final denotes B10, except for one Amazon–Kimi trajectory whose original endpoint is B9; that result is retained unchanged.

We further extend an original Yelp–DeepSeek trajectory beyond B10 to B50. At each budget, the program is selected by validation performance and then evaluated on test:

Discovery budget BValidation NDCG@10Test NDCG@10
100.0559650.049673
200.0567610.050747
300.0569700.051175
500.0581830.054767

Increasing the budget from B10 to B50 improves validation NDCG@10 from 0.055965 to 0.058183, while test NDCG@10 also increases from 0.049673 to 0.054767 (+10.26%). Thus, the additional improvements selected on validation continue to transfer to test beyond the original search budget.

These observations provide evidence that the measured gains are not confined to the reused validation set, without assuming that aggregate feedback eliminates adaptive selection bias.

W2 and Q2 (a): One-shot generation with the same LLM

We compare one-shot generation with ProgramRec, both using DeepSeek-V4-Pro. One-shot (B1) uses the first valid program from each original trajectory, constructed before receiving validation feedback from any previous candidate. Here, one-shot denotes one complete program-construction round, including target-free execution checks and repair.

The table reports test NDCG@10, as mean ± sample standard deviation over the three original trajectories. Both the B1 and B10 results retain the original frozen-program evaluations.

DatasetOne-shot (B1)ProgramRec (B10)
Amazon Movies & TV0.0203 ± 0.00620.0427 ± 0.0064
Goodreads0.0537 ± 0.00840.1128 ± 0.0072
Yelp0.0426 ± 0.00210.0484 ± 0.0014

ProgramRec achieves higher mean test NDCG@10 on all three datasets, demonstrating gains from continued discovery beyond the initial program. We next examine the role of cross-round feedback while holding the evaluated-program budget fixed.

W2 and Q2 (b): No-history comparison and the role of cross-round feedback

For the additional no-history baseline, DeepSeek-V4-Pro independently constructs ten valid programs without exposing previous candidate sources or validation outcomes to subsequent proposals. The final program is still selected by validation NDCG@10. This baseline reuses the original ProgramRec-discovered features and frozen recommendation stack, and retains within-candidate checking and repair. ProgramRec instead uses the accumulated program sources and aggregate validation feedback to guide subsequent proposals. Both methods evaluate ten valid programs.

The table reports test NDCG@10, as mean ± sample standard deviation over three trajectories per method. ProgramRec retains its original results; no-history uses the additional experiments.

DatasetNo-history (B10)ProgramRec (B10)
Amazon Movies & TV0.0207 ± 0.00120.0427 ± 0.0064
Goodreads0.0697 ± 0.01210.1128 ± 0.0072
Yelp0.0454 ± 0.00030.0484 ± 0.0014

At the same budget of ten evaluated programs, ProgramRec achieves higher mean test NDCG@10 on all three datasets. This supports the contribution of accumulated source–outcome feedback beyond repeated independent generation and validation-based selection alone.

W2: Actual discovery cost

We complement the discovery-call counts in Appendix A.3.6 (Table 14) with cumulative input/output token usage and the corresponding test performance. B counts formally evaluated programs, whereas constructing each program can require multiple LLM calls. We count completed formal-search calls, including normal candidate construction and within-candidate checking and repair. Costs are accumulated through the specified budget endpoint.

To relate performance to actual discovery cost as the budget grows, we report the cumulative costs of the same original Yelp–DeepSeek trajectory extended to B50 in our response to W1/Q1. M denotes one million tokens.

BLLM callsInput tokens (M)Output tokens (M)Total tokens (M)Test NDCG@10
101082.2200.1202.3400.049673
202064.7650.2655.0300.050747
303017.4550.4167.8710.051175
5047713.1320.68813.8200.054767

Along this trajectory, cumulative token usage increases from 2.340M to 13.820M, while test NDCG@10 increases from 0.049673 to 0.054767 (+10.26%). These checkpoints therefore express the observed scaling behavior in terms of actual LLM usage as well as the number of evaluated programs. The selected program is subsequently reused without further coding-LLM calls, so this generation cost is incurred during discovery.

Dear Reviewer,

We're following up on our response. As the discussion period is nearing its end, we'd be grateful for an update on whether it addresses your concerns, or if any further clarification is needed.

Thank you for your time.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.