- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an active learning method for fine-tuning a UNET based video world model called EnerVerse-AC (EVAC), which is trained on robotics data. Authors propose to train a confidence predictor on top of the frozen model. This confidence map is then used to select episodes comprising the most difficult examples (lowest confidence), and for increasing weights on most difficult frames and patches at training time. Experiments on a new robot dataset show some performance gains compared to the original model, as well as the model trained on a randomly sample dataset.
Critical strengths
- Actively selecting the most relevant samples with a fixed budget is a great experiment towards making better use of compute and training data, in situations were budgets are restricted. Choosing samples where models are most uncertain could be a valuable strategy.
- The confidence estimation procedure is sensible, and experiments suggest some gains to the sample selection approach.
- Authors carried a very large set of experiments to evaluate their approach in great detail, included results across multiple seeds, and comparison to several state of the art models.
Critical weaknesses
- The confidence computation section in 3.1 essentially follows the methodology described in the C3 paper (Mei et al, 2026). The main differences claimed in the introduction are the use of a U-Net vs a DiT (the main needed change here seems to be deciding where embeddings are taken) and an EMA-based threshold supervision (which shows no clear advantage in experiments). While the introduction does mention that confidence estimation follows C3, the original work should be credited accurately in the methods section, and the core changes required should be highlighted clearly.
- As a result, the novelty is limited: the main innovation relates to leveraging confidence information for pre-selection of training samples, which is a narrower contribution. Results show incremental gains compared to a EVAC v1 model and frame/patch weighting yield mixed results. For example, frame weighting tends to have worse results than no weighting at all.
- I have concerns over using confidence as a way of measuring episode difficulty or areas to focus training on. This does not seem to take into account potential stochastic behaviours, which could lead to low confidence if multiple outcomes are equally valid, yet will be valid predictions. Isn’t there a risk to over learn on specific outcomes (especially with a fixed selection budget) and obtain poorly calibrated models?
- In section 4.2, authors use pixel/latent error as a ground truth reference to evaluate their confidence/risk prediction. However, those measure two separate quantities (thinking again about stochastic events), and while pixel error can be useful for context, it should not be used as a ground truth for confidence. I am not sure that this section answers the “why confidence?” question, but rather verifies that confidence estimation is sensible.
- Minor: The related work could be structured better, with different approaches grouped together with no clear connections.
Questions for authors
- I find it surprising that the Random baseline does worse than EVAC v1, since it is trained on a larger dataset. Is there a particular selection process for the v1 dataset? Is the assumption that training the conference probe in parallel helps?
How were training budgets chosen? 6,248 episodes for v1 and 7,298 for retraining seems very specific. It would also be good to understand how the approach compared to a model finetuned on the full dataset. Is there a remaining gap? Is focusing on hard examples only better? - Are stochastic events taken into account in the method design? Have authors verified impact of their approach on world model calibration?
- Sample selection is carried out with a different model than the one eventually evaluated. Why not add new samples during training instead of doing this in two stages? This could also enable curriculum type approaches.
The response has resolved all of my concerns. I am therefore raising my score.