- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents an ablation study of visual reward model training, comparing four training objectives using ~2k training examples:
- BCE 2. BCE + Invariance Loss 3. BCE + Sensitivity Loss 4. BCE + Invariance + Sensitivity. The experiments show that BCE + Sensitivity Loss achieves the best offline validation performance. However, the authors do not evaluate every variant in downstream RL. I think this paper‘s novelty is quite low and experiments lack information.
Critical strengths
- The paper provides a controlled comparison of different visual reward model training objectives using the same training data.
- Paper is easy to understand.
Critical weaknesses
- Lack of Novelty: The paper primarily combines well-established techniques, such as BCE, invariance regularization, and sensitivity-based ranking losses, and conducts ablation studies on their combinations. I did not gain any substantial new insights from the experiments.
- Limited Training Data: The paper uses only 600 original trajectories, augmented into 2,229 training examples, to train the visual reward models. This dataset is relatively small, especially for drawing general conclusions about visual reward model training.
- Unconvincing Conclusions: I find the overall conclusions of the paper unconvincing. The experiments are conducted on a relatively small dataset, and the incomplete downstream ablations make it difficult to draw meaningful conclusions about the effectiveness of different training objectives.
- Poor Figure Quality: The figures seem to be generated by AI and lack professional polish.
Questions for authors
- Incomplete Downstream Ablations: Why was the BCE + Sensitivity Loss variant, which achieved the best offline validation performance, not evaluated in downstream RL?
- Limited Training Data: The reward models are trained using only 600 original trajectories augmented into 2,229 examples. How sensitive are the conclusions to the training data scale?