- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies RL in which only a noisy sum of rewards is observed over each feedback segment, . TrustMix combines a tabular ridge reference with structured surrogates, which are either fitted to the same labels or fixed priors (expert, LLM). At each state–action pair, the weights are simplex weights that minimize . Here is a joint sandwich/bootstrap covariance. The bias proxies come from variance-corrected disagreement with the reference, shifted by a ridge-bias bound scaled by . The minimized proxy also serves as an exploration bonus. Prop 2 bounds the oracle-weight MSE by . Thm 1 gives regret against the best stationary policy for an idealized tabular procedure. Experiments cover an advertiser MDP, a sepsis simulator, and FSRS-Exam (SAC with four LLM-derived surrogates).
Critical strengths
Bagged reward setting is an important and practical setting
The proofs are careful. I checked the main steps and found no errors.
Empirical results seem strong
Critical weaknesses
The setting is not new. Eq. 1 is the bagged-reward setting (Tang et al., TMLR 2025), and it is close to bagged decision times (Gao et al., AISTATS 2025). Neither is cited. The novelty claims should be narrowed.
The key solution: mixing a set of reward surrogate is quite simple, and I don't see clear connection with the bagged setting. The bagged reward setting seems orthogonal to the main methodology contribution.
I don't get the point of regret bound in Thm1. I was hoping to see discussions about how quality of the surrogate rewards affect regret. For example, if one of the surrogate reward is the true reward, one can significantly improve the bound.
Assumption 1 is unrealistic. A single noise scale applies to all labels, yet longer segments should be noisier.
I don't understand why we need assumption 2. It is a very strong assumption. This point requires discussion.
The theory does not match the algorithm. The comparator is restricted to stationary policies, and Algorithm 1's value iteration does not satisfy Assumption 3. hides , and factors.
The theory does not match the algorithm. The comparator is restricted to stationary policies, and Algorithm 1's value iteration does not satisfy Assumption 3.
The empirical gains are early and small. Reference-only catches up in the advertiser MDP. The no-prior variant ends better in sepsis. In FSRS-Exam, the gain over round-robin has a CI that includes 0, and the weights stay near uniform.
Questions for authors
-
How does TrustMix relate to bagged-reward RL (Tang et al., 2025) and bagged decision times (Gao et al., 2025)? Which part of the method is specific to aggregate feedback?
-
Can Assumption 2 be removed?