acceptodds
Under review as a conference paper at ICLR 2027

Reweight, Do Not Re-calibrate: Reusing the Same Encoder-Predictor Pair (World Model) When the Policy or Query Changes

Abstract

A world model calibrated under one policy is queried under another once a planner selects actions from its predictions. We ask whether the same frozen encoder-predictor can be reused under a changed policy or a changed query law without re-calibration on target labels. Conformal off-policy prediction marginalizes the action and predicts a return, which makes the importance weight a function of the outcome and forces an outcome model, rejection subsampling, or a trajectory product. We condition on the action and predict the next state, which makes the weight measurable before the outcome and removes all three; the reduction holds at one step, and the occupancy factor returns over a horizon. With a planner enacting the new policy on RoboCasa, exact propensity weighting of the source calibration set reaches 0.913 coverage with no new observations, against 0.910 after recalibrating on 75 labelled target episodes per task (model seed 271828, temperature 3); recalibration re-collects 300 labelled episodes under each new policy, while the weighted construction reuses one 600-episode source set across all nine target policies. We prove that the target-to-source likelihood ratio is measurable before the outcome and factors into state-occupancy and action-propensity ratios, and combine weighted conformal prediction with the exact or an estimated ratio. Under a query change, coverage at nominal 0.90 moves from 0.902 to 0.550 and to 0.909 with learned weights on 192 Franka transitions, and from 0.734 to 0.903 on 5,100 FurnitureBench episodes, matching oracle weighting; with V-JEPA-2-AC in place of DINOv2 on the Franka data, 0.561 and 0.917. Under the planner-enacted policy change, coverage moves from 0.750 to 0.912 with each radius finite, a paired gain of 16.2 points [8.5, 23.7], replicating on two further model seeds at 0.690 to 0.877 and 0.735 to 0.892. For a learned ratio, the exact convex dual of the classifier-regret constraint within each quantile bin of the posterior gives coverage lower bounds of 0.863 and 0.802 on the two query-change corpora, against observed 0.909 and 0.903. Finite-support identities of the argument are machine-checked in Lean 4.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, uncertainty quantification, distribution shift, covariate shift, density ratio estimation, world models, robot learning, off-policy evaluation, calibration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LJ6GOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper appears to deal with calibration of a world model - if the model was trained on data collected by some policy, can its prediction be trusted when rolled out under a different policy/planner? The authors study this question, and validate their findings on a real-world Franka setup, and on simulated tasks in Robocasa. Unfortunately, I am not able to provide further details. Despite spending a significant amount of time on this submission, I find it very hard to parse, both in its theoretical and experimental parts.

Critical strengths

Due to presentation, I am not able to point out critical strengths.

Critical weaknesses

The main weakness of this submission is presentation. The entirety of the manuscript appears heavily LLM generated, and it is very hard to read. A few examples:

  • The abstract is abrupt, it does not introduce what the submission is about, and instead uses very technical language for no apparent reason, e.g. “propensity weighting of the source calibration set”, “the exact convex dual of the classifier-regret constraint within each quantile bin of the posterior gives coverage lower bounds”. The majority of these technical terms have not been introduced, and unless the reader is extremely familiar with this work, they are likely to be not informative.
  • The abstract reports detailed experimental results, which is not useful at this point. I would advise reporting the overall conclusion in an easy to process way, and leaving most numbers for later.
  • Some LLMy speech patterns repeat over and over, e.g. “it’x not X, but Y”.
  • The introduction reads as a list of statement, without a specific definition of what the goal of this paper is, what the contributions are nor how it is organized. It introduces some equations early on, without defining all variables involved. Suddenly, it switches to numerical results and parameters, without a clear explanation of why this level of detail is already necessary.
  • Several sentences are exceedingly long and complicated, e.g. on line 104, which states that one question will be asked, and proceeds to ask three of them.
  • Background/preliminaries are missing or scattered, while they would be very beneficial.
  • The introduction is followed by three theoretical sections that I was unable to follow, as the ultimate goal and its meaningfulness are not clearly stated.
  • The experimental section presents a lot of numbers directly in the text, but does not draw conclusions from them.
  • Related works and limitations are missing, or scattered.
  • The titles do not match, and there is a format violation in the subtitle.

I am unable to confidently point out other weaknesses as a consequence of the main one.

Questions for authors

I do not have questions that could materially affect my assessment. I do not think I would be able to recommend acceptance unless the paper was fundamentally rewritten to improve clarity, and in that case I would need to evaluate its contribution.

As a suggestion, I would ask:

  • Can the authors provide a gentle, non-technical introduction to the contributions of the paper, followed by a rigorous mathematical description of them? Adding a background section would really help?
  • Can the authors reorganize the experiments such that the text has fewer numerical details, and also conveys the overall conclusion from each experiment?
  • Can the authors shift towards a human and understandable language through the work?

Note: I will keep my confidence low as it is possible that I lack fundamental background which other reviewers may have.

Reviewer_cCagOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how to ureuse its old prediction errors to estimate uncertainty in the case that the way in which it chooses its actions changes. It proposes a way to reuse those uncertainty estimates along with the new predictions to calibrate its uncertainty using conformal prediction. In simple terms this has the effect of making the old examples that are similar to the new data be weighted more. The authors evaluate their method in a couple of different benchmarks including Franka Robot, RoboBench and RoboCasa. Where they demonstrate that despite the data changing substantially in nature, the uncertainty estimators retain their performance.

Critical strengths

The paper proposes the use of a statistically sound method for uncertainty recalibration in robotics and shows the method works well in a couple of robotic benchmarks. This is an important problem as it shows that such technique can be used to reduce the sample needs of a robot during deployment.

The evaluation included real robot data, and was generally upfront about the evaluation limitations. This is highly appreciated. For example the authors do comment on how RobotCasa and FurnitureBench both do not change the state distribution, and thus these benchmarks cannot test for corrections in then occupancy measure during execution of a test policy.

Critical weaknesses

My main concern is the limited novelty of the contribution. The statistical techniques are not novel in the literature. The work is thus of the type "used X from Y in domain Z". It is a well executed implementation but it is unclear if this passes the bar for a publication at ICLR.

As mentioned above, some of the experiments by design do not explore the case of occupancy change, so it is hard to establish significance from the empirical results.

Questions for authors

Can the authors clarify what do they consider to be their main technical innovations in this work?

Do you have any comment about the underrepresentation of scenarios testing for robustness to state occupancy shifts?

Reviewer_TupzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Completely nonsensical AI slop. The authors should be ashamed of themselves for this blatant misconduct.

Critical strengths

None.

Critical weaknesses

This paper is complete nonsense. A few examples below:

  • L46: "an over-confident prediction set becomes a collision rather than a bad number"
  • L47: "A capable agent cannot avoid carrying such a model"
  • L63: "On the Franka corpus"
  • L104: "We ask one question: under an intervention on action selection, with the conditional transition mechanism invariant, which change of measure transports marginal coverage from source to target, how much coverage do we lose when we estimate it, and when do unlabeled target covariates recover the guarantee?"
  • L137: "We use P1 to separate prediction from calibration."
  • L150: "When we intervene at one step from a common state population, dt = db and the propensity ratio is the whole correction."
  • L154: "Deployed decision systems log the probability of each chosen action at decision time (Agarwal et al., 2016; Swaminathan & Joachims, 2015; Gauci et al., 2018; Saito et al., 2021), and we read the source density from such a log."
Questions for authors
  • What drove you to submit AI slop to ICLR?
  • What are you trying to achieve?
  • Do you feel ashamed?
  • Why do you think this is OK?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.