- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces OCBench, a controllable benchmark for studying behavioral cloning with synthetic trajectories designed to resemble human demonstrations. It reproduces several empirical phenomena seen in real-world BC, including overfitting improving performance, action chunking outperforming closed-loop control, the need for large policies, and sensitivity to input features. The paper also uses controlled interventions to study possible causes of these effects.
Critical strengths
The paper studies several important but poorly understood BC phenomena in a controlled setting. I especially liked the experiments on action chunking and Markovianized data, which make the benchmark more useful than just another task suite.
Critical weaknesses
The main weakness is that the “human-like” claim is not fully validated. The paper shows similar action statistics and reproduces similar phenomena, but does not directly compare richer trajectory-level properties with real human demonstrations. Also, several properties of the scripted policy are changed together, so it is sometimes unclear which one is responsible for each phenomenon.
Questions for authors
- Can the authors provide a more direct comparison between OCBench and human demonstrations, especially on temporal correlation or trajectory-level diversity?
- Have the authors separately ablated the main design choices of the scripted policy to see which ones are necessary for reproducing each phenomenon?
2 comments
- Have the authors separately ablated the main design choices of the scripted policy to see which ones are necessary for reproducing each phenomenon?
Thanks for the question. We'd like to mention two points. First, to understand why open-loop control is better than closed-loop control, we performed a controlled ablation where we replaced temporally correlated trajectories with Markovianized trajectories in Figure 8, while keeping everything else the same. The result shows that the gap between closed- and open-loop policies narrows, suggesting that Phenomenon 2 can be partially (but not fully) explained by the non-Markovianity of the dataset. This is an example of how OCBench's tunable knobs enable controlled ablation studies. Second, the difference between full and lite variants provides a complementary comparison. The lite variant removes gripper pitch/roll randomization and shortens the effective horizon. As a result, these lite tasks can be learned with smaller networks (1024-sized as opposed to 4096-sized for the full version). During development, we found that individual changes from full to lite enable us to use smaller networks. This suggests that the complexity of demonstrations contributes to the large-model requirement discussed in Phenomenon 3.
We would like to thank you again for raising important questions about our work. We believe the additional results and clarifications have strengthened the paper. Please let us know if you have any additional concerns or questions. If we have addressed your concerns, we would appreciate it if you could adjust the score accordingly.
Thank you for the detailed review and constructive feedback on this work. We especially appreciate your comments about human-likeness. Following your suggestion, we have measured three quantitative metrics to show how OCBench has more similar structural properties to human demonstrations and clarified other points. Please find our response below.
Thanks for raising this point. We'd like to first clarify that, by "human-like", we meant OCBench trajectories have similar structural properties to human demonstrations (e.g., temporally correlated, smooth, etc.). We didn't intend to claim that the synthetic trajectories themselves are necessarily very similar to human demonstrations, which is a much more subjective and harder-to-justify claim. While we tried to convey this point throughout the paper (e.g., L44, Sections 2-3), we found that some phrases might potentially be interpreted in the other way, and we've adjusted wording (changes in red) and further clarified this point in L139.
Then, regarding whether OCBench really has more similar structural properties to human demonstrations than previous scripted policies, we would like to share results from two perspectives.
First, we measured three quantitative metrics for trajectory-level properties, (1) action saturation (the one used in the paper), (2) speed consistency (measuring how much trajectories are halting/phase-y), and (3) turning concentration (how "sharp" turning movements are), hypothesizing that they capture some structural properties (e.g., smoothness) of human demonstrations. We measured them on OCBench, previous synthetic benchmarks, and human demonstrations.
(More details: For action saturation, we measure the fraction of actions that have ∣a∣≥0.95 after normalization, where asterisks mean we measure them after converting to joint deltas. For speed consistency, we measure the ratio of speed at bends to speed on nearby portions (within 2.5cm) of the trajectory. For turning concentration, we measure the fraction of directional change concentrated in the (sharpest) 10% of path locations within 10cm windows.)
The table above compares the three metrics of OCBench, two human-generated datasets and six previous scripted or policy-generated datasets. The results support that OCBench has more similar properties than other synthetic datasets in general. While these metrics aren't perfect (they are proxies), we believe these metrics (partly) support our claim about structural similarity.
Second, we also provide a complementary video comparison between OCBench, human demonstrations, and previous synthetic datasets at https://ocbench-videos.pages.dev/. The videos show that OCBench trajectories are much smoother, less "phase-y"/mechanistic, and more diverse (e.g., random tilts, retries, etc.) than the previous ones (even though OCBench trajectories may not be perfectly "human-like"). While this is a qualitative comparison, it provides a complementary perspective on why OCBench and previous benchmarks may have different metrics in the above table.