acceptodds
Withdrawn from ICLR 2027

Universe-as-Code: Executable Asset Selection for Portfolio Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Portfolio reinforcement learning depends on both the assets available to a policy and the reward used to train it. We propose UCAPS (Universe-as-Code Agent Portfolio System), which learns executable upstream design artefacts for these coupled decisions. Universe-as-Code (UaC) generates multiple asset-selection programs, validates them in a sandbox, and combines their outputs through deterministic per-ticker voting. Dual-Artefact Co-Evolution (DACE) evaluates universe–reward pairs through downstream policy training and uses successful outputs to update separate Selector and Coder adapters. Its Composite Robust Score (CoRS) aggregates multi-seed feedback on historical performance, drawdown, temporal dispersion, and program complexity. On a 30-ticker multi-asset pool, UCAPS achieves 44.1% mean Test cumulative return and 1.31 Sharpe across four Selector backbones and 80 policy runs, net of transaction costs. The mean return exceeds the strongest reported same-pool large language model (LLM) agent adaptation by 10.5 percentage points. Price-matched classical baselines supply a common data reference, and eight matched selection controls isolate the selection rule. Under a fixed reference reward, the Qwen2.5-7B and Llama-3.1-8B Selectors exceed the strongest alternative by 20.6 and 11.1 percentage points. These results establish executable universe selection as an effective upstream optimization target for portfolio reinforcement learning.

Primary area:
reinforcement learning
Keywords:
portfolio reinforcement learning, universe selection, code-generating LLMs, reward design, co-evolution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DhoLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UCAPS, a framework that uses LLMs to generate executable asset-selection programs. The framework evaluates asset-universe and reward pairs through downstream PPO training and uses the results to fine-tune the code generators. Experiments on an asset pool report improvements over classical portfolio strategies, RL baselines, and LLM-based trading agents. The authors also provide ablation studies and analyses of different model backbones, asset pools, and re-selection frequencies.

Critical strengths

The paper is well-written. The paper treats executable asset-selection programs as an optimization target rather than directly selecting assets with LLMs. Using downstream RL performance to improve the program generator is an interesting approach to portfolio optimization.

The paper evaluates multiple Selector backbones and compares against a broad range of baselines. The matched selection experiments and component ablations provide useful evidence for the effectiveness of the proposed framework.

Critical weaknesses

The paper lacks a direct comparison with a simple best-of-N approach that samples multiple asset universes, trains PPO for each, and selects the best based on calibration performance. The random-universe baseline reports average performance over 50 draws rather than selecting the best candidate. A comparison under matched computational budgets is important to establish whether the proposed Selector fine-tuning and co-evolution framework provide benefits beyond straightforward search.

The main experiments rely on a single 30-asset pool and one test period. It remains unclear whether the reported improvements generalize to different market periods and asset universes.

Questions for authors

Can the authors compare UCAPS against a best-of-N universe search baseline using the same PPO evaluation budget and calibration protocol? This would help determine whether Selector fine-tuning provides benefits beyond searching over more candidate universes.

Can the authors provide additional out-of-sample evaluations across different market periods or asset pools using the full UCAPS pipeline? This would help assess the generalizability of the reported improvements.

The reference to Forging Better Rewards is listed as Anonymous (2025). Since the authors' names have now been disclosed, could the authors update the citation?

The R&D-Agent-Quant reference lists seven authors, whereas the official NeurIPS proceedings version lists six. Could the authors correct the citation?

Reviewer_dWT8OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UCAPS uses LLM-generated programs to select an investable asset universe and construct rewards for a downstream portfolio policy. Selection programs are checked in a sandbox, and their proposed subsets are combined by ticker-level voting. The DACE loop evaluates universe–reward pairs through PPO training and uses successful examples to update separate Selector and Coder adapters. A composite score combines historical performance, drawdown, temporal dispersion, and reward-code length. The main evaluation uses a 30-asset pool, historical calibration through December 2024, and a January 2025–March 2026 test period. The paper reports higher cumulative returns than its baseline implementations and includes fixed-reward selection comparisons, component ablations, and deployment diagnostics.

Critical strengths

The fixed-reward selection experiment is the strongest evidence in the paper. Holding the reward and PPO stack fixed, the Qwen Selector reaches 47.2% cumulative return, compared with 26.6% for the symbolic scorer. This comparison is more informative than the heterogeneous agent leaderboard. The authors also report that simple handwritten rewards on the selected universe reach 47.0–48.5%, close to the co-evolved reward’s 48.8%. Reporting this result helps identify where the system’s benefit actually comes from.

Critical weaknesses

The evaluation establishes a favorable selected portfolio on one market path, but does not yet establish the proposed learning method.

The headline 80 runs consist of repeated PPO training for four fixed deployment universes under a shared reward. They measure downstream training variability conditional on those choices. They do not measure variability across independently learned Selectors, independent DACE searches, or different market deployment periods. The narrow seed-mean intervals therefore cannot support a similarly precise conclusion about the selection procedure’s future performance. All four universes also contain AAPL, GLD, and PLTR, so the backbone comparison offers limited diversification of the underlying investment decision.

The supplementary studies do not resolve this issue. The “cross-pool” equity and ETF pools are subsets of the original candidate pool and use the same Test period. The temporal slices reuse the same deployed configurations and market trajectory. The 2023–2024 cadence study is explicitly a retrospective replay of the retained programs, rather than a fresh chronological evaluation of learning and selection. These are useful diagnostics, but not independent replications of the main claim. I do not infer Test leakage from this; the problem is insufficient evidence of generalization.

The selection controls also leave open a simpler explanation for the gain. DACE searches against downstream portfolio feedback, whereas the learned scoring baselines are trained on forward 63-session Sharpe or selected by rank information coefficient. Matching input statistics and PPO training does not match the upstream optimization objective or search budget. The example executable rule in Appendix D is a compact combination of Sharpe and drawdown with an eligibility threshold. A direct search over such scoring formulas, thresholds, or subsets using the same historical evaluator is a necessary comparator.

This matters because the paper’s contribution is the method for learning executable selection rules, rather than the identification of one successful five-asset portfolio. The current results do not sufficiently distinguish those outcomes.

Questions for authors
  1. Are there independent end-to-end DACE replications for the main Selector, including new program sampling and adapter updates? Please report the resulting universe variation and Test outcomes separately from PPO-seed variation. More importantly, can the full procedure be evaluated at multiple historical deployment boundaries, with every search and selection decision restricted to information available at each boundary?

  2. How does UCAPS compare with a non-LLM formula or subset search given the same downstream PPO evaluator, calibration objective, and evaluation budget? A useful control would search the Sharpe–momentum–drawdown formula family, including the eligibility filter. The large drop when removing that filter makes this comparison particularly consequential.

  3. Please quantify sensitivity to an execution model that charges turnover relative to drifted pre-trade holdings. Equation (2) charges changes between consecutive target vectors; an unchanged target vector can still require trading after relative asset prices move. Please also include a daily classical comparator under the same execution convention. These checks would help establish how much of the reported advantage survives more closely aligned trading assumptions.

Reviewer_87qLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UCAPS generates executable asset-selection programs and reward code for portfolio PPO. Validated selection programs vote on a five-asset universe, while historical PPO feedback guides updates to the selector and reward coder. The reported mean test return is 44.1% across 80 PPO runs.

Critical strengths
  1. The matched selection experiment is persuasive. With the reward and PPO protocol fixed, the Qwen-based selector returns 47.2%, compared with 26.6% for symbolic regression and at most 18.4% for the fixed ranking rules.

  2. Executable selection rules are an interesting, inspectable way to specify the investable universe. The paper also reports useful ablations and re-selection diagnostics.

Critical weaknesses
  1. The main result comes from one curated 30-asset pool and one test period. The 80 runs measure PPO seed variation, not variation across market periods or asset pools. The reported cross-pool result is mixed: top-five selection underperforms the full ETF pool.

  2. The evidence for joint co-evolution is weaker than the evidence for selection. On the final universe, the co-evolved reward returns 48.8%, versus 48.5% for a simple handwritten reward.

  3. A simple allocator on the exact deployed five-asset universe, under the same cost and trading schedule, would help separate the value of selecting assets from the value of PPO allocation.

Questions for authors
  1. How stable are the selected assets and returns across rolling train/calibration/test splits?

  2. What do equal-weight and risk-parity allocation achieve on the deployed universe under the same execution protocol?

  3. Selector updates give each valid program credit for the ensemble’s result. How often do individual programs disagree with the voted universe, and does contribution-weighted replay change the outcome?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.