- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper appears to deal with calibration of a world model - if the model was trained on data collected by some policy, can its prediction be trusted when rolled out under a different policy/planner? The authors study this question, and validate their findings on a real-world Franka setup, and on simulated tasks in Robocasa. Unfortunately, I am not able to provide further details. Despite spending a significant amount of time on this submission, I find it very hard to parse, both in its theoretical and experimental parts.
Critical strengths
Due to presentation, I am not able to point out critical strengths.
Critical weaknesses
The main weakness of this submission is presentation. The entirety of the manuscript appears heavily LLM generated, and it is very hard to read. A few examples:
- The abstract is abrupt, it does not introduce what the submission is about, and instead uses very technical language for no apparent reason, e.g. “propensity weighting of the source calibration set”, “the exact convex dual of the classifier-regret constraint within each quantile bin of the posterior gives coverage lower bounds”. The majority of these technical terms have not been introduced, and unless the reader is extremely familiar with this work, they are likely to be not informative.
- The abstract reports detailed experimental results, which is not useful at this point. I would advise reporting the overall conclusion in an easy to process way, and leaving most numbers for later.
- Some LLMy speech patterns repeat over and over, e.g. “it’x not X, but Y”.
- The introduction reads as a list of statement, without a specific definition of what the goal of this paper is, what the contributions are nor how it is organized. It introduces some equations early on, without defining all variables involved. Suddenly, it switches to numerical results and parameters, without a clear explanation of why this level of detail is already necessary.
- Several sentences are exceedingly long and complicated, e.g. on line 104, which states that one question will be asked, and proceeds to ask three of them.
- Background/preliminaries are missing or scattered, while they would be very beneficial.
- The introduction is followed by three theoretical sections that I was unable to follow, as the ultimate goal and its meaningfulness are not clearly stated.
- The experimental section presents a lot of numbers directly in the text, but does not draw conclusions from them.
- Related works and limitations are missing, or scattered.
- The titles do not match, and there is a format violation in the subtitle.
I am unable to confidently point out other weaknesses as a consequence of the main one.
Questions for authors
I do not have questions that could materially affect my assessment. I do not think I would be able to recommend acceptance unless the paper was fundamentally rewritten to improve clarity, and in that case I would need to evaluate its contribution.
As a suggestion, I would ask:
- Can the authors provide a gentle, non-technical introduction to the contributions of the paper, followed by a rigorous mathematical description of them? Adding a background section would really help?
- Can the authors reorganize the experiments such that the text has fewer numerical details, and also conveys the overall conclusion from each experiment?
- Can the authors shift towards a human and understandable language through the work?
Note: I will keep my confidence low as it is possible that I lack fundamental background which other reviewers may have.