acceptodds
Under review as a conference paper at ICLR 2027

Sparse Activation Attributes for Frame Curation and Curriculum Scheduling in Imitation Learning

Abstract

Training vision-language-action (VLA) models requires large-scale robot demonstration datasets, making data curation critical for both data efficiency and policy performance. Existing approaches typically select entire episodes or coarse temporal blocks and sample the retained data from a fixed distribution, leaving substantial frame-level redundancy and overlooking how the value of data with different attributes evolves throughout training. We introduce Sparse Activation Attributes-inspired Frame Curation and Curriculum Scheduling strategy (SAFari), a fine-grained framework that uses sparse activation attributes to jointly select training frames and schedule their presentation. SAFari first employs a sparse autoencoder to extract intermediate activation features and characterizes them by their generality and specificity. It then uses the strengths of these two attributes to assess individual frames and independently constructs general and specific frame streams. Finally, a step-wise sigmoid curriculum dynamically shifts the sampling emphasis from general to specific frames as training progresses. In this way, SAFari not only prioritizes informative frames at a fine-grained level, but also presents data with different attributes at appropriate stages of training through curriculum scheduling. Experiments across multiple VLA models in both simulation and real-robot settings demonstrate consistent performance gains. Moreover, SAFari maintains strong performance with only half the training steps, substantially improving training efficiency.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Imitation Learning, Data Curation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_okixOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAFari, a data curation and curriculum learning framework for robot imitation learning. It extracts intermediate representations from a frozen Octo-Small policy and trains a BatchTopK sparse autoencoder (SAE) to obtain sparse features. Feature generality and specificity are estimated using episode coverage and sustained activation duration, with a logistic classifier trained on a small set of manually labeled features. The resulting feature attributes are aggregated into two frame-level scores, which are used to construct partially overlapping general and specific data streams. A sigmoid-controlled curriculum gradually shifts the sampling distribution from general to specific frames during training. The method is evaluated on LIBERO-90 with Octo-Small, π₀, and OpenVLA, as well as on four real-robot tasks using xArm6.

Critical strengths
  1. Fine-grained data curation is a relevant problem. The paper addresses temporal redundancy in robot demonstrations by selecting individual frames rather than entire episodes or coarse temporal chunks. Combining frame selection with a training-time curriculum is a practical and potentially useful design.

  2. The empirical evaluation provides encouraging evidence. The authors evaluate three policy architectures and four real-robot tasks. The ablations on frame selection strategies, general/specific streams, and curriculum ordering help demonstrate that multiple components contribute to the reported improvements. The reduced-training-budget results also suggest potential gains in data and optimization efficiency.

Critical weaknesses
  1. The selection of representations and implementation details have not been sufficiently studied. The method uses the frozen Octo-Small representation for data filtering, but it cannot prove that Octo is the most suitable source of representations for data filtering. The paper does not compare the intermediate representations of the target strategy itself (such as pi0 and OpenVLA used in this paper) or other visual encoders (such as DINO), and the author does not explain clearly how multi-view observations are transformed into frame-level representations in the 11th layer of Octo.

  2. The estimation of generality and specificity lacks sufficient verification. The classifier merely used 20 manually selected features and validated them on this small-scale dataset. This result cannot prove that the classifier can generalize to the remaining SAE features. Moreover, the method only estimated the two attributes based on two heuristic statistics and defined specificity as the complement of generality while the authors acknowledged that the two are not strictly mutually exclusive. Finally, 1000 out of 1536 features were retained, and there was also a lack of sensitivity analysis regarding the number of features or classification thresholds.

  3. The choice of methods is not innovative. The main experiments in the article still use Octo and OpenVLA. These model structures are rather outdated and have insufficient pre-training. It is necessary to cover models such as OpenVLA-OFT and pi05, which have undergone extensive pre-training.

Questions for authors
  1. The foundation of the article is based on the premise that the intermediate representations generated by the frozen Octo can contain sufficient feature information. Is this premise valid? Can models such as pi0/pi05 pre-trained with larger-scale data obtain better intermediate representations? Can such intermediate layer representations achieve a one-to-one correspondence with the raw data frame by frame? Can the author elaborate on how to obtain the intermediate representations of the 11th layer of Octo for multi-view data?

  2. The classifier was trained only on the 20 manually labeled feature dimensions. Is this level of labeling sufficient? Can the author provide evidence to show that this mapping can generalize to randomly selected or independently labeled features? Is retaining 1000 features more empirically advantageous compared to significantly fewer or more features?

  3. Is training a binary classifier based on a small number of 0/1 labels to generate continuous scores reasonable? The article has already stated that general and specific are not absolutely complementary, but is it reasonable that the scores of the two add up to 1?

  4. Could the author distinguish the impact brought about by the representation of the frozen Octo from the impact of the frame filtering and course scheduling mechanism itself? Could the author conduct ablation experiments on the representation using intermediate representations (pi0, OpenVLA) of different target strategies or other visual encoders (DINO)?

1 comment

Thank you for the detailed responses and additional experiments. The new comparisons provide convincing evidence for the effectiveness of SAE-based selection and address my concerns regarding data budget fairness. The results under matched frame retention rates further strengthen the paper. Although some details regarding the construction of SAFari-50% and the individual contributions of selection and curriculum scheduling could be clarified further, I do not consider these issues major concerns. I maintain my original rating of Weak Accept.

Reviewer_uCNfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAFari, a data curation and curriculum learning framework for robot imitation learning. It uses SAE to extract the features in the hidden state from a pretrained VLA model to characterize general and specific information, selects two corresponding frame streams, and gradually shifts training from general to specific samples. Experiments on simluation and real-world manipulation tasks demonstrate improvements across baselines including Octo, π0, and OpenVLA.

Critical strengths
  1. The paper presents an interesting approach that leverages SAE-derived feature attributes for data curation and curriculum learning in robot imitation learning and the framework is straightforward to implement and does not require modifications to the policy architecture or training objective, making it easy to integrate with different robot learning policies.

  2. The method achieves substantial improvements across multiple VLA models and shows encouraging results in simluated and real-world robotic manipulation tasks.

Critical weaknesses
  1. The necessity of SAE. While the ablations demonstrate the effectiveness of the proposed scoring and curriculum components, they do not isolate the contribution of SAE representations. Comparisons with simpler alternatives, such as dense pretrained features, visual changes, or action-based importance scores, are missing. Consequently, it remains unclear whether SAE-based attribute decomposition is essential to the observed improvements.

  2. Data budgets. According to Appendix C, competing methods retain 50% of demonstrations, whereas SAFari retains 50% of frames in each of two streams. Appendix B.4 shows that their union covers 76.88% of the original frames. This difference in selection granularity and unique-frame coverage complicates the interpretation of the performance advantage over existing curation methods.

Questions for authors
  1. Can the authors compare SAFari against curriculum strategies constructed using dense VLA representations or simpler visual/action-based scores, under matched training budgets?

  2. Can the authors compare SAFari with existing curation methods under matched unique-frame retention rates to better isolate the contribution of the proposed selection strategy?

Reviewer_Q6xaOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SAFari, a method that selects and samples training frames for robot imitation learning. The approach trains a sparse autoencoder using hidden representations from a frozen Octo-Small model. It then uses a classifier to estimate how general or specific each sparse feature is, based on episode coverage and activation duration. The resulting scores are used to divide frames into two groups: general and specific. During policy training, a sigmoid schedule gradually shifts sampling from general frames to specific frames. The authors test SAFari with Octo-Small, π0, and OpenVLA in simulation, and with π0 on four real-robot tasks. They report that policies trained on SAFari-selected frames achieve higher task success rates than those trained on the full dataset or frames selected by competing methods. SAFari also achieves higher success rates with half the policy training steps compared with full-data training using the complete training budget.

Critical strengths

S1. The paper addresses a practical problem by considering two questions together: which frames to train on and when to use them. It keeps the original action targets and policy loss unchanged and modifies only the data pipeline. This makes the method easier to apply to existing policies. For example, the SAE trained on Octo-Small is reused for π0 and OpenVLA.

S2. The method works at the frame level, which is finer than the episode-level or block-level selection in the baselines.

S3. The authors test three VLA models (Octo-Small, π0, and OpenVLA) in simulation, and π0 on four real-robot tasks.

Critical weaknesses

Major:

W1. The comparison to the baselines does not control the total quantity of training data (Section 4.1, Appendix B.4, Figure 7). Each baseline keeps the top 50 percent of the episodes. The paper does not report the exact number of frames this equals. In contrast, SAFari keeps 514,389 of 669,043 frames, which is a total unique retention of 76.88 percent (Appendix B.4). SAFari therefore trains on about 1.5 times more unique frames than the baselines. Furthermore, the random control in Figure 7 appears to keeps only 50 percent of the frames. The paper does not have a random control that matches the 76.88 percent budget of SAFari.

W2. The methods differ in task coverage, which introduces a confounding variable (Section 4.1). The baselines remove whole episodes. Because the LIBERO-90 dataset has 90 different tasks, removing whole episodes means the baselines might completely delete all demonstrations for some tasks. In contrast, SAFari selects individual frames from all episodes. This allows SAFari to keep data for every task. Therefore, SAFari might outperform the baselines simply because it does not forget entire tasks. To prove the selection method is better, the baselines and SAFari must have the same task coverage.

W3. The benefit of SAE-based scoring over simpler selection signals is unclear (Section 3.1, Figure 7). Figure 7 compares attribute-score ranking with fixed-interval selection. The success rates are 62.3 and 56.1 percent. This supports score-based selection, but it does not isolate the benefit of the SAE-derived attributes. The paper does not compare them with simpler frame scores, such as action change or embedding novelty. A comparison with the same frame budget would show the added value of SAE-based scoring.

Minor:

The efficiency claim needs a measured cost breakdown (Section 4.4, Table 1). Table 1 excludes the offline data curation. Section 4.4 calls this cost "negligible" but does not give a duration. The full pipeline includes activation extraction, SAE training, feature scoring, frame selection, and manual feature labeling. A new domain needs new SAE training and new labels. The time per step is the same as for all data, so the saving comes only from the lower number of steps. Please report the time for each step and compare it with the offline cost of the baselines.

Figure 1 has “Course Scheduling” and Figure 2 has “Pipline”. The first contribution bullet starts with a lowercase “we”.

Questions for authors

Q1. How many frames does each baseline keep? How many demonstrations of each of the nine evaluation tasks remain after each baseline selects its episodes? If some tasks lose most or all of their demonstrations, the baseline comparison is not fair.

Q2. What is the result of a random control that keeps 76.88 percent of the frames, with uniform sampling and no schedule? Please compare it with the "No curriculum" result in Figure 7 (70.8 percent). What are the results when the baselines use the same frame budget and a task-balanced selection? If SAFari still wins, I will raise my soundness score.

Q3. What are the retention and the success rate for each evaluation task? How much of the gain comes from the difference in task balance between SAFari and the baselines?

Q4. Does a simple frame score, such as action change or embedding novelty, give a similar result? Please use the same frame budget (76.88 percent unique frames) and compare with the "No curriculum" result in Figure 7 (70.8 percent).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Revisions

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. Our main contributions are:

  • We derive generality and specificity attributes from sparse feature activations to identify high-value training frames.
  • We combine redundancy-aware frame selection with a general-to-specific curriculum to improve policy performance and training efficiency.
  • We improves performance across architectures and real-robot tasks, achieving 78.2% success and 66.2% with half the training steps.

To further support these contributions, we have added experiments and technical clarifications, summarized below for easy reference:

  • Data budgets and task coverage: Matched-budget comparisons and task-stratified controls (Fig. 3; Appendices B.3–B.5).
  • SAE effectiveness: Comparisons against three simpler frame-scoring signals (Appendix B.5).
  • Representation choice: Alternative representation sources and multi-view extraction details (Appendices A.1 and B.6).
  • Feature classification: Independent validation, feature-count ablation, and clarification of attribute definitions (Section 3.1; Appendices A.2–A.3).
  • Curation efficiency: Cost breakdown and baseline comparisons (Appendix B.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.