acceptodds
Under review as a conference paper at ICLR 2027

TrustMix: Reward Learning from Aggregate Feedback with Heterogeneous Surrogates

Abstract

When reward feedback in reinforcement learning arrives sparsely or only as a delayed aggregate, a learner may observe whether a trajectory succeeded without knowing which decisions produced the outcome. This weak credit assignment leaves local rewards poorly identified, particularly early in learning. Expert knowledge and LLM-derived reward priors can provide valuable structure, but they may be biased, with reliability varying across the state–action space. We introduce TrustMix, which combines a data-driven reference model with heterogeneous structured surrogate reward models and allocates trust locally among them. Surrogates generalize to weakly observed decisions, while the reference uses observed outcomes to calibrate trust in potentially misspecified priors as evidence accumulates. We characterize when complementary surrogates improve reward estimation and provide a sublinear-regret guarantee in an idealized tabular setting. Experiments across three environments show that TrustMix improves early learning while adapting its local weights as evidence accumulates.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, aggregate feedback, delayed rewards, sparse rewards, credit assignment, reward learning, surrogate reward models, inductive bias, large language models, Human-AI collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bjodOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies RL in which only a noisy sum of rewards is observed over each feedback segment, y=Xr⋆+εy = Xr^\star + \varepsilon. TrustMix combines a tabular ridge reference with MM structured surrogates, which are either fitted to the same labels or fixed priors (expert, LLM). At each state–action pair, the weights are simplex weights that minimize w⊤(Σ^+b^b^⊤)ww^\top(\hat\Sigma + \hat b\hat b^\top)w. Here Σ^\hat\Sigma is a joint sandwich/bootstrap covariance. The bias proxies b^\hat b come from variance-corrected disagreement with the reference, shifted by a ridge-bias bound BjB_j scaled by βB∈[−1,1]\beta_B\in[-1,1]. The minimized proxy also serves as an exploration bonus. Prop 2 bounds the oracle-weight MSE by min⁡{R00, bˉ2+σs2(ρ+(1−ρ)/M)}\min\{R_{00},\ \bar b^2+\sigma_s^2(\rho+(1-\rho)/M)\}. Thm 1 gives O~(N)\tilde O(\sqrt N) regret against the best stationary policy for an idealized tabular procedure. Experiments cover an advertiser MDP, a sepsis simulator, and FSRS-Exam (SAC with four LLM-derived surrogates).

Critical strengths

Bagged reward setting is an important and practical setting

The proofs are careful. I checked the main steps and found no errors.

Empirical results seem strong

Critical weaknesses

The setting is not new. Eq. 1 is the bagged-reward setting (Tang et al., TMLR 2025), and it is close to bagged decision times (Gao et al., AISTATS 2025). Neither is cited. The novelty claims should be narrowed.

The key solution: mixing a set of reward surrogate is quite simple, and I don't see clear connection with the bagged setting. The bagged reward setting seems orthogonal to the main methodology contribution.

I don't get the point of regret bound in Thm1. I was hoping to see discussions about how quality of the surrogate rewards affect regret. For example, if one of the surrogate reward is the true reward, one can significantly improve the bound.

Assumption 1 is unrealistic. A single noise scale σ\sigma applies to all labels, yet longer segments should be noisier.

I don't understand why we need assumption 2. It is a very strong assumption. This point requires discussion.

The theory does not match the algorithm. The comparator is restricted to stationary policies, and Algorithm 1's value iteration does not satisfy Assumption 3. CNC_N hides SA\sqrt{SA}, HH and 1/p1/\sqrt p factors.

The theory does not match the algorithm. The comparator is restricted to stationary policies, and Algorithm 1's value iteration does not satisfy Assumption 3.

The empirical gains are early and small. Reference-only catches up in the advertiser MDP. The no-prior variant ends better in sepsis. In FSRS-Exam, the gain over round-robin has a CI that includes 0, and the weights stay near uniform.

Questions for authors
  1. How does TrustMix relate to bagged-reward RL (Tang et al., 2025) and bagged decision times (Gao et al., 2025)? Which part of the method is specific to aggregate feedback?

  2. Can Assumption 2 be removed?

Reviewer_XJWQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reinforcement learning from trajectory- or segment-level aggregate feedback, where local rewards are not directly observed. TrustMix combines a flexible, data-driven reference reward model with structured surrogates, including learned models and fixed expert or LLM-derived priors. It selects state–action-specific convex weights by minimizing an estimated MSE objective based on joint sampling covariance and reference-based bias proxies. The paper provides conditions for improved oracle reward estimation and a sublinear-regret guarantee for an idealized tabular procedure, and reports early-learning improvements in advertising, sepsis-treatment, and study-scheduling simulators.

Critical strengths
  1. Combining useful but potentially biased reward knowledge is particularly relevant when aggregate feedback provides weak local supervision. The method explicitly accounts for both location-dependent reliability and correlations between models fitted to the same labels, rather than treating all reward predictions as independent or equally reliable.

  2. Beyond the oracle MSE result, the paper establishes a stability bound for the actual data-dependent mixture without requiring the covariance estimate to equal the true covariance. The authors also explicitly distinguish the resulting regret guarantee from a learning-speed improvement and discuss its limitations.

  3. The experiments report early-learning gains across three simulation tasks using 50 matched seeds or user profiles and uncertainty estimates. The Advertiser experiment includes deliberately misspecified surrogates and a uniform-weight ablation, while FSRS-EXAM separates tuning profiles from held-out evaluation profiles.

Critical weaknesses
  1. Although Advertiser includes uniform weighting, Sepsis and FSRS-EXAM lack comparably informative controls using the same priors, such as prior-only policies, fixed mixtures, or a global adaptive mixture. Consequently, the improvements may primarily reflect access to useful structured reward knowledge rather than the proposed local trust allocation.

  2. Proposition 2 establishes benefits for oracle weights, whereas Theorem 1 preserves a regret rate rather than guaranteeing improvement over reference-only learning. The latter requires uniform holding probabilities, a stationary-policy comparator, and specified planning and exploration conditions; no reported experiment fully matches this setup, and the theorem does not directly guarantee the performance of the final bonus-free policy.

  3. The weight plots mainly show visitation-weighted averages, whose evolution can reflect changes in visited states as well as changes in local weights. These plots do not establish whether the method assigns greater weight to locally more accurate models, especially when the reference itself is unreliable, and the reported ablations do not isolate the covariance and bias-correction components.

  4. The evaluation uses task-specific surrogate constructions without systematically varying their bias, redundancy, or number, leaving robustness to predominantly misleading priors unclear. Moreover, the numerical choices and selection procedure for βB\beta_B are not reported, and sensitivity to this parameter and the fixed neural reference-bias allowance is not established.

Questions for authors
  1. Could the authors compare against single-surrogate and fixed-mixture baselines using the same priors, particularly in Sepsis and FSRS-EXAM? A global adaptive mixture or a simple surrogate-to-reference switching schedule would further clarify whether state–action-specific weights are necessary.

  2. In Advertiser, where the true reward is available for evaluation, could the authors assess weights and bias proxies on a fixed set of state–action pairs? Ablating cross-model covariance and variance-corrected disagreement would help identify which components contribute to the gains.

  3. Could the authors provide a diagnostic experiment satisfying Theorem 1’s setup, including its policy class and exploration specification? Please distinguish the collection-policy regret covered by the theorem from the performance of the bonus-free evaluation policy.

  4. Please report the βB\beta_B values and selection procedure for each experiment, together with sensitivity to βB\beta_B and the continuous reference-bias allowance. How strongly do these choices affect the observed transition from surrogate reliance to reference reliance?

  5. What happens when useful surrogates are supplemented with strongly biased or near-duplicate models, or when the number of surrogates increases? This would help establish whether the method handles heterogeneous reliability beyond the particular pools used in the experiments.

Reviewer_YRPwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies reinforcement learning from trajectory or segment level feedback when local rewards are unobserved. TrustMix fits a flexible ridge reference to aggregate labels and combines it with structured reward surrogates, some learned from the same labels and others supplied as fixed priors. At each state-action pair, it estimates joint predictive covariance and uses variance-corrected disagreement with the reference plus a reference-bias allowance to form a local MSE proxy, and simplex weights minimize this proxy. Mixed rewards and uncertainty bonuses guide data collection. The experiments cover a continuing advertiser MDP, a synthetic sepsis simulator, and a real-user-calibrated FSRS study-scheduling simulator with LLM-derived priors. They report early performance gains over reference-only and delayed-feedback baselines.

Critical strengths
  1. The combination problem is well motivated. The paper recognizes that several reward models trained on the same aggregate labels have correlated errors and that a reference can itself be biased in poorly identified directions (Sec. 3.2). This is more careful than treating surrogate errors as independent or the reference as ground truth.

  2. The evaluation is broad and reasonably transparent. TrustMix improves early policy quality in the advertiser and sepsis tasks and beats the learned FSRS-EXAM comparators on the paired held-out profiles. The paper also states the substantial limits of its regret result in Appendix C.14, which helps readers assess what is actually established.

Critical weaknesses
  1. The method’s novelty needs sharper support. Local expert gating and bias-variance pooling are established ideas. The distinctive contribution is the reference-disagreement bias proxy under shared aggregate feedback, but its incremental value is not isolated.

  2. The core weighting rule lacks decisive ablation. Only the advertiser study compares uniform weights, and the sepsis and FSRS-EXAM lack matched fixed or global-weight controls. FSRS-EXAM also reports nearly equal mean LLM weights, so gains may come from the priors themselves.

  3. Theorem 1 covers collection-policy regret for an idealized additive tabular procedure, while every experiment evaluates a final bonus-free policy outside that guarantee. The authors acknowledge this scope, but the theorem does not establish robustness of the implemented policies.

Questions for authors
  1. Using the same reward models, online budgets, planner/SAC settings, and tuning protocol within each task, what are the paired policy-performance differences between TrustMix and (i) equal local weights and (ii) a tuned global reference-plus-surrogate mixture in sepsis and FSRS-EXAM? A bias-proxy ablation would further isolate the contribution of Eq. 9–11 from the benefit of having informative surrogates.

  2. Given the explicitly stated scope of Theorem 1, can you report a theorem-aligned experiment with its prescribed coefficients and collection-policy regret, or establish a result for the final bonus-free policy under conditions closer to the implementations? This would show how much of the theoretical assurance transfers to the policies evaluated in the paper.

  3. The paper appropriately treats the fitted rewards in sepsis and FSRS-EXAM as policy-useful pseudo-rewards rather than identified primitive rewards. How well do their aggregate predictions and the resulting true policy values generalize to trajectories induced by policies different from those used to fit the reward model?

  4. How would a simple non-LLM, state-aware ranking prior built from the same learner-visible features perform if converted to a frozen surrogate through the same architecture and combined with TrustMix under the same online budget? This would isolate the incremental value of LLM-derived priors.

  5. In the advertiser MDP, where the true local reward is known, how well do the estimated local bias and MSE proxies track actual error as coverage changes, including when surrogates have shared or opposite-sign bias? Please state the β_B values used and show sensitivity to them and, for FSRS-EXAM, to its 0.05 reference-bias allowance. In sepsis, what is the paired uncertainty on the 600-episode full-versus-no-prior policy-gap difference?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.