acceptodds
Under review as a conference paper at ICLR 2027

What Predictive-Improvement Rewards Make Agents Collect

Abstract

Predictive-improvement rewards value an observation by how much it helps predict later measurements. An agent that chooses those measurements can spend its budget making earlier discoveries earn credit. We study what this incentive makes it collect. With enough independent noiseless bits and an even budget, every policy maximizing the complete observation-deletion reward queries each chosen bit twice and acquires half the attainable information. An exact frontier characterizes the tradeoff, and a matching formulation shows how it depends on scoring times. The preference can also redirect learned behavior: switching an effective Gaussian-field policy to deletion reward increases credit, reduces coverage and leaves 45% more reconstruction error than continuing task training. Fitting masked predictions to policy records adds another dependence. Retained actions reveal deleted outcomes, changing the reward even with exact predictions; refitting can cycle or sustain a self-consistent optimum with lower information. Finally, interventions on temperature records show how unavailable inputs and changing evaluation targets alter the task value of selected measurements. These results connect the choices used to assign predictive credit to the information a finite acquisition budget buys.

Primary area:
reinforcement learning
Keywords:
intrinsic motivation, exploration, predictive improvement, information gain, active sensing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_U3YWOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work does a systematic characterization about the problems of learning with observation-deletion rewards. Through several examples and environments including GP, Mastermind, and temperature sensor records, the authors analyze the induced policies from such rewards, and importantly, how such polices are suboptimal for the underlying tasks.

Critical strengths

This work is insightful, and the authors did a great job of supporting their arguments with both theory and experiments.

Critical weaknesses

To me, the main takeaway from this paper is that observation-deletion type of reward is not ideal, and the authors did a solid work in demonstrating this. However, it is unclear to me to how broadly such type of reward is used, and additionally, can the related insights extend other forms of rewards.

Questions for authors

How can the findings guide us to a practical reward design? This is a somewhat future-work kind of question, so a broad answer would be fine.

Reviewer_ktR3OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks what predictive-improvement rewards, which credit an observation by how much deleting it hurts later predictions, actually make an agent collect under a finite budget. With noiseless independent bits, every reward-maximizing adaptive policy queries each chosen bit twice and gets only half the attainable information. The authors characterize the full reward/information frontier and extend the result to general time weights via maximum matchings. They also show that when the masked predictor is fit to policy data, retained actions leak the deleted outcomes, so refitting can cycle or settle at a self-consistent point that collects less information. GP, Mastermind and Intel sensor experiments back these claims.

Critical strengths
  1. Propositions 1 and 2 characterize reward-optimal policies over all adaptive policies, together with their information cost and the exact frontier. These are sharp, checkable statements rather than empirical trends.
  2. The distinction between interventional and observational masked predictors is useful. Proposition 3's strict cycle and the 21-point identification gap in Mastermind show that action leakage matters in practice for anyone training predictors on policy data.
Critical weaknesses
  1. It is unclear which existing methods actually optimize Eq. (1). By Eq. (3), the fixed-design reward is the dual total correlation, so a preference for redundant observations is partly expected. The largest information loss appears only in the noiseless independent-bit setting. In Appendix J.5, the description-length objective of Elmoznino et al. reaches essentially the same MSE as information gain on the GP task (.1923 vs .1930). So I am not sure how much this failure mode affects methods people use.
  2. The Intel experiment in Section 6 is only loosely connected to the main argument. It mostly reflects the selector's assumptions about missing data. The mean reversal is driven by a few tail days: at four requests, dropping March 25 flips the sign of the gap. This section takes main-text space without strengthening the core claim.
Questions for authors
  1. Which published methods optimize Eq. (1) or a weighted version of it? With a learned predictor, do methods such as NDIGO still prefer paired confirmation?
  2. When the observation model is unknown, is there a design that keeps a predictive-improvement reward but avoids the bias toward confirmation? The probe variants in J.5 still fall well short of information gain on the GP task.
Reviewer_2DPEOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

At a high level, this paper studies whether rewarding observations that improve future predictions encourages useful information collection. The paper attempts to understand when observation-deletion rewards encourage agents to collect new information and when they instead favor repeated measurements. It also examines how policy-dependent predictions and missing sensor reports affect which measurements are preferred and how useful they are for the task.

The paper is very to read and understand, so my summary is incomplete and not detailed.

Critical strengths

The paper attempts to highlight a distinction between maximizing predictive reward and collecting useful information. The paper attempts to explain the main message through some case studies and illustrations. I did not understand most of the details in the paper to be able to appreciate its strengths well.

Critical weaknesses

Overall, I found the paper very difficult to parse. The approach and experimental setups need clearer explanations of what is being tested, how each experiment is conducted, and what we should take away from the results.

The main text also needs a clearer discussion of prediction error and the key hypothesis the paper is trying to test. The paper should distinguish failures caused by the reward formulation from those caused by errors in the predictors.

it is unclear how Section 6 supports the these of the paper. The sensor results depend on missing reports, which dates are included in the evaluation, target availability, and model error. Although the paper includes comparisons, it needs to state more directly the role of this section.

The figures are not informative and should be revised to complement the text.

Finally, the scope of the paper in terms of what downstream systems are being targetted is unclear. Also the evidence for more general learning systems is limited.

Questions for authors

Please refer to the weaknesses. Can the paper be re-written for clarity to make it possible to understand the claims and results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.