- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SAFari, a data curation and curriculum learning framework for robot imitation learning. It extracts intermediate representations from a frozen Octo-Small policy and trains a BatchTopK sparse autoencoder (SAE) to obtain sparse features. Feature generality and specificity are estimated using episode coverage and sustained activation duration, with a logistic classifier trained on a small set of manually labeled features. The resulting feature attributes are aggregated into two frame-level scores, which are used to construct partially overlapping general and specific data streams. A sigmoid-controlled curriculum gradually shifts the sampling distribution from general to specific frames during training. The method is evaluated on LIBERO-90 with Octo-Small, π₀, and OpenVLA, as well as on four real-robot tasks using xArm6.
Critical strengths
-
Fine-grained data curation is a relevant problem. The paper addresses temporal redundancy in robot demonstrations by selecting individual frames rather than entire episodes or coarse temporal chunks. Combining frame selection with a training-time curriculum is a practical and potentially useful design.
-
The empirical evaluation provides encouraging evidence. The authors evaluate three policy architectures and four real-robot tasks. The ablations on frame selection strategies, general/specific streams, and curriculum ordering help demonstrate that multiple components contribute to the reported improvements. The reduced-training-budget results also suggest potential gains in data and optimization efficiency.
Critical weaknesses
-
The selection of representations and implementation details have not been sufficiently studied. The method uses the frozen Octo-Small representation for data filtering, but it cannot prove that Octo is the most suitable source of representations for data filtering. The paper does not compare the intermediate representations of the target strategy itself (such as pi0 and OpenVLA used in this paper) or other visual encoders (such as DINO), and the author does not explain clearly how multi-view observations are transformed into frame-level representations in the 11th layer of Octo.
-
The estimation of generality and specificity lacks sufficient verification. The classifier merely used 20 manually selected features and validated them on this small-scale dataset. This result cannot prove that the classifier can generalize to the remaining SAE features. Moreover, the method only estimated the two attributes based on two heuristic statistics and defined specificity as the complement of generality while the authors acknowledged that the two are not strictly mutually exclusive. Finally, 1000 out of 1536 features were retained, and there was also a lack of sensitivity analysis regarding the number of features or classification thresholds.
-
The choice of methods is not innovative. The main experiments in the article still use Octo and OpenVLA. These model structures are rather outdated and have insufficient pre-training. It is necessary to cover models such as OpenVLA-OFT and pi05, which have undergone extensive pre-training.
Questions for authors
-
The foundation of the article is based on the premise that the intermediate representations generated by the frozen Octo can contain sufficient feature information. Is this premise valid? Can models such as pi0/pi05 pre-trained with larger-scale data obtain better intermediate representations? Can such intermediate layer representations achieve a one-to-one correspondence with the raw data frame by frame? Can the author elaborate on how to obtain the intermediate representations of the 11th layer of Octo for multi-view data?
-
The classifier was trained only on the 20 manually labeled feature dimensions. Is this level of labeling sufficient? Can the author provide evidence to show that this mapping can generalize to randomly selected or independently labeled features? Is retaining 1000 features more empirically advantageous compared to significantly fewer or more features?
-
Is training a binary classifier based on a small number of 0/1 labels to generate continuous scores reasonable? The article has already stated that general and specific are not absolutely complementary, but is it reasonable that the scores of the two add up to 1?
-
Could the author distinguish the impact brought about by the representation of the frozen Octo from the impact of the frame filtering and course scheduling mechanism itself? Could the author conduct ablation experiments on the representation using intermediate representations (pi0, OpenVLA) of different target strategies or other visual encoders (DINO)?
Thank you for the detailed responses and additional experiments. The new comparisons provide convincing evidence for the effectiveness of SAE-based selection and address my concerns regarding data budget fairness. The results under matched frame retention rates further strengthen the paper. Although some details regarding the construction of SAFari-50% and the individual contributions of selection and curriculum scheduling could be clarified further, I do not consider these issues major concerns. I maintain my original rating of Weak Accept.