- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the behavior of in-context RL when exposed to held-out tasks from outside the convex hull of training support. They elect to estimate the agent’s implicit task belief and find saturation outside the training “band.” They use decision-transformer (and its variants) along with some meta-RL context encoders. This saturation is hypothesized to be caused by the authors' proposed "convex aggregation". Using an extra hand-tuned statistic channel, they find mixed results in success, depending on whether it’s ID or OOD, but find better extrapolation.
Critical strengths
- I can appreciate the selection of the problem here. I do believe isolating a model’s latent inference of a task could be a good analytical tool or pathway for understanding in-context generalization.
Critical weaknesses
- A critical weakness of this paper is that there is an unclear contribution with a poorly written delivery. The introduction reads like a summary without a clear description of why one should care about the topic and how it relates to broader impact. Some parts of the paper are nearly unreadable such as the Discussion section. The experimental section itself is filled with numerical details and sparse takeaways and insights for the community.
- Another weakness is the scope of the paper which may be too narrow for a conference main track submission. This further cements how the submission is incomplete and not fit for acceptance. From what I can understand from the discussion: “we recommend also reporting the parameter region covered in training and each evaluation task’s position relative to it.” The paper as it stands could perhaps be condensed into a couple pages or so, and a strong method for overcoming the issues of generalization of task inference could be studied in detail.
- Lastly, the results of this paper don’t seem particularly conclusive. The paper primarily discusses HalfCheetah-Vel in the results, with no solid reporting of other tasks (e.g., in Figures 1 and 2). Many things are also marked as exploratory.
Writing nitpicks:
- In the abstract, “requested outward change” is unspecified. Generally, the abstract should contain higher-level details, but this one seems to use jargon which would be elucidated only by reading the paper itself. Reading onwards, it seems like it’s properly covered around like 147-9.
- The first paragraph reads like a related works section. It’d be much nicer to be introduced to the motivation and issues at hand from a bird’s eye view and get a hint at your key insights.
- What does line 50 mean, I am not sure what “contaminated” refers to.
- Continuing the read the intro, I feel strongly that it is not calibrated as an intro but seemingly a kind of “rapid summary” of the paper itself. - I’d appreciate if the introduction was more clear about the gravity of this problem, the issues that the current style of thinking has caused, the desiderata, and the core insight you’d like to share with the community.
- By Section 3, I’m still a bit confused on what the work’s contributions are and what I’m in for when I’m reading. For the most part, there seem to be very “appendix-like” details embedded in the first glance of what your method/contribution’s section will show.
- Not sure why on lins 209-15 there’s a dump of numerical data; it’s a bit hard to parse and probably doesn’t belong in text. To be honest, a majority of page 5 is like this too. Perhaps using the text to communicate the main message and keeping visualizations to communicate numbers would be a better policy
- The results continue to read quite overloaded with jargon, numerical details, and very low-level observations.
- The discussion is nearly incomprehensible with very unclear contributions.
Questions for authors
None
Thank you for your response!
After giving the paper another read, it is still quite hard to parse and not very clearly written. Regarding scope, I can understand this is a measurement paper, but the measurement is not particularly groundbreaking if the takeaway is: “we recommend also reporting the parameter region covered in training and each evaluation task’s position relative to it." Generally, my concerns don't feel resolved.
For these reasons, I will keep my score. In its current state, the paper needs significant work and should be resubmitted with a more insightful story and conclusive set of results.