acceptodds
Under review as a conference paper at ICLR 2027

OCBench: A Controllable Benchmark for Studying Behavioral Cloning

Abstract

Behavioral cloning (BC), despite its simplicity, exhibits many counterintuitive phenomena in the real world. For example, the performance of BC often keeps increasing as the model overfits more to the dataset, and fully closed-loop policies often completely fail without action chunking. Unfortunately, properly studying these empirical phenomena is challenging: in the real world, datasets and experiments are costly and not fully controllable; in simulation with synthetic data, these phenomena are often not easily observed partly due to the discrepancy between scripted policies and human demonstrations. In this work, we propose OCBench, a robotic manipulation benchmark with controllable scripted policies that have similar properties to human demonstrations. We show that, by mimicking key properties of human demonstrations, OCBench reproduces many anecdotal BC-related phenomena in controlled settings. With its GPU-accelerated environments and scripted policies, we demonstrate how OCBench enables scientific studies of previously reported BC-related phenomena by analyzing and refuting various hypotheses.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotics, behavioral cloning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_mE3HOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OCBench, a controllable benchmark for studying behavioral cloning with synthetic trajectories designed to resemble human demonstrations. It reproduces several empirical phenomena seen in real-world BC, including overfitting improving performance, action chunking outperforming closed-loop control, the need for large policies, and sensitivity to input features. The paper also uses controlled interventions to study possible causes of these effects.

Critical strengths

The paper studies several important but poorly understood BC phenomena in a controlled setting. I especially liked the experiments on action chunking and Markovianized data, which make the benchmark more useful than just another task suite.

Critical weaknesses

The main weakness is that the “human-like” claim is not fully validated. The paper shows similar action statistics and reproduces similar phenomena, but does not directly compare richer trajectory-level properties with real human demonstrations. Also, several properties of the scripted policy are changed together, so it is sometimes unclear which one is responsible for each phenomenon.

Questions for authors
  1. Can the authors provide a more direct comparison between OCBench and human demonstrations, especially on temporal correlation or trajectory-level diversity?
  2. Have the authors separately ablated the main design choices of the scripted policy to see which ones are necessary for reproducing each phenomenon?
2 comments

Thank you for the detailed review and constructive feedback on this work. We especially appreciate your comments about human-likeness. Following your suggestion, we have measured three quantitative metrics to show how OCBench has more similar structural properties to human demonstrations and clarified other points. Please find our response below.

  • "The main weakness is that the “human-like” claim is not fully validated. The paper shows similar action statistics and reproduces similar phenomena, but does not directly compare richer trajectory-level properties with real human demonstrations." / "Can the authors provide a more direct comparison between OCBench and human demonstrations, especially on temporal correlation or trajectory-level diversity?"

Thanks for raising this point. We'd like to first clarify that, by "human-like", we meant OCBench trajectories have similar structural properties to human demonstrations (e.g., temporally correlated, smooth, etc.). We didn't intend to claim that the synthetic trajectories themselves are necessarily very similar to human demonstrations, which is a much more subjective and harder-to-justify claim. While we tried to convey this point throughout the paper (e.g., L44, Sections 2-3), we found that some phrases might potentially be interpreted in the other way, and we've adjusted wording (changes in red) and further clarified this point in L139.

Then, regarding whether OCBench really has more similar structural properties to human demonstrations than previous scripted policies, we would like to share results from two perspectives.

First, we measured three quantitative metrics for trajectory-level properties, (1) action saturation (the one used in the paper), (2) speed consistency (measuring how much trajectories are halting/phase-y), and (3) turning concentration (how "sharp" turning movements are), hypothesizing that they capture some structural properties (e.g., smoothness) of human demonstrations. We measured them on OCBench, previous synthetic benchmarks, and human demonstrations.

(More details: For action saturation, we measure the fraction of actions that have ∣a∣≥0.95|a| \geq 0.95 after normalization, where asterisks mean we measure them after converting to joint deltas. For speed consistency, we measure the ratio of speed at bends to speed on nearby portions (within 2.5cm) of the trajectory. For turning concentration, we measure the fraction of directional change concentrated in the (sharpest) 10% of path locations within 10cm windows.)

DatasetTaskAction saturation (%)Speed consistency (%)Turning concentration (%)
Human demo (Robomimic PH)lift\texttt{lift}1.6767.231.9
Human demo (BiPlay)play\texttt{play}1.13*73.825.0
OCBench (ours)block-double-task1\texttt{block-double-task1}0.2278.631.8
OGBenchcube-double-play\texttt{cube-double-play}11.1689.631.7
ManiSkill3 MPStackCube\texttt{StackCube}1.37*32.852.3
ManiSkill3 PPOStackCube\texttt{StackCube}5.8369.929.5
RoboTwin 2.0\texttt{stack_blocks_two}1.46*33.732.0
RLBench\texttt{stack_blocks}3.00*35.159.4
Robomimic MGlift\texttt{lift}19.0468.728.2

The table above compares the three metrics of OCBench, two human-generated datasets and six previous scripted or policy-generated datasets. The results support that OCBench has more similar properties than other synthetic datasets in general. While these metrics aren't perfect (they are proxies), we believe these metrics (partly) support our claim about structural similarity.

Second, we also provide a complementary video comparison between OCBench, human demonstrations, and previous synthetic datasets at https://ocbench-videos.pages.dev/. The videos show that OCBench trajectories are much smoother, less "phase-y"/mechanistic, and more diverse (e.g., random tilts, retries, etc.) than the previous ones (even though OCBench trajectories may not be perfectly "human-like"). While this is a qualitative comparison, it provides a complementary perspective on why OCBench and previous benchmarks may have different metrics in the above table.

  • Have the authors separately ablated the main design choices of the scripted policy to see which ones are necessary for reproducing each phenomenon?

Thanks for the question. We'd like to mention two points. First, to understand why open-loop control is better than closed-loop control, we performed a controlled ablation where we replaced temporally correlated trajectories with Markovianized trajectories in Figure 8, while keeping everything else the same. The result shows that the gap between closed- and open-loop policies narrows, suggesting that Phenomenon 2 can be partially (but not fully) explained by the non-Markovianity of the dataset. This is an example of how OCBench's tunable knobs enable controlled ablation studies. Second, the difference between full and lite variants provides a complementary comparison. The lite variant removes gripper pitch/roll randomization and shortens the effective horizon. As a result, these lite tasks can be learned with smaller networks (1024-sized as opposed to 4096-sized for the full version). During development, we found that individual changes from full to lite enable us to use smaller networks. This suggests that the complexity of demonstrations contributes to the large-model requirement discussed in Phenomenon 3.


We would like to thank you again for raising important questions about our work. We believe the additional results and clarifications have strengthened the paper. Please let us know if you have any additional concerns or questions. If we have addressed your concerns, we would appreciate it if you could adjust the score accordingly.

Reviewer_BLuyOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • The authors created a new benchmark dataset, OCBench to analyze various phenomena of behavioral cloning. In particular, they study BC overfitting, open-loop vs closed-loop, policy size, and and input features. In each of these, they replicate findings from previous works and analyze hypotheses for each
  • The authors conduct preliminary experiments providing evidence that this dataset produces more human-like demonstrations than a selection of prior works.
Critical strengths
  • Unlike many other prior works, the dataset is GPU optimized, leading to more efficient data generation allowing for large-scale analyses across many robotic tasks and environments
Critical weaknesses
  • The evidence presented that the dataset is more human-like is not compelling. Figure 1 only compares against two other previous datasets, and only uses one metric of human-likeness - qualitative distribution similarity
  • While the authors were able to demonstrate that OCBench is able to serve as a test bed for investigating these four phenomena, they do not provide any theoretical or intuition analysis showing why this formulation of the task and dataset is particularly suited towards this analysis, compared to other constructions. Moreover, the authors do not provide empirical evidence that other datasets (e.g. OGBench, Robomimic, MimicGen, RoboCasa, RLBench, or others) are unsuited for analyzing these or other phenomena. In particular, it would be valuable to show that for each finding, OCBench is able to replicate the finding or disprove the hypotheses, when none of the other datasets allow for this.
Questions for authors
  • Do you have additional evidence or justifications for calling OCBench more humanlike than prior works (including many other prior works which were mentioned in the related work but not tested empirically)?
  • Do you have any examples of phenomena or hypotheses which only OCBench is capable of demonstrating or rejecting, while every other dataset fails at doing these?
5 comments

Thank you for the detailed review and constructive feedback on this work. We especially appreciate your comments about human-likeness and how OCBench is different from previous synthetic datasets. Following your suggestion, we have measured three quantitative metrics to show how OCBench has more similar structural properties to human demonstrations, conducted comparisons with three additional previous synthetic datasets, and revised several phrases in the paper. Please find our response below.

  • "Do you have additional evidence or justifications for calling OCBench more humanlike than prior works (including many other prior works which were mentioned in the related work but not tested empirically)?"

Thanks for raising this point. We'd like to first clarify that, by "human-like", we meant OCBench trajectories have similar structural properties to human demonstrations (e.g., temporally correlated, smooth, etc.). We didn't intend to claim that the synthetic trajectories themselves are necessarily very similar to human demonstrations, which is a much more subjective and harder-to-justify claim. While we tried to convey this point throughout the paper (e.g., L44, Sections 2-3), we found that some phrases might potentially be interpreted in the other way, and we've adjusted wording (changes in red) and further clarified this point in L139.

Then, you might still ask whether OCBench really has more similar structural properties to human demonstrations than previous scripted policies. While this is still a somewhat subjective claim (by nature), we would like to defend it. First, please watch this video comparison at https://ocbench-videos.pages.dev/, where we compare OCBench with scripted/policy-generated trajectories from 6 different benchmarks as well as human demonstrations. The videos show that OCBench trajectories are much smoother, less "phase-y"/mechanistic, and more diverse (e.g., random tilts, retries, etc.) than the previous ones (even though OCBench trajectories may not be perfectly "human-like").

We further support this point quantitatively. We measured three quantitative metrics: (1) action saturation (the one used in the paper), (2) speed consistency (measuring how much trajectories are halting/phase-y), and (3) turning concentration (how "sharp" turning movements are), hypothesizing that they capture some structural properties (e.g., smoothness) of human demonstrations.

DatasetTaskAction saturation (%)Speed consistency (%)Turning concentration (%)
Human demo (Robomimic PH)lift\texttt{lift}1.6767.231.9
Human demo (BiPlay)play\texttt{play}1.13*73.825.0
OCBench (ours)block-double-task1\texttt{block-double-task1}0.2278.631.8
OGBenchcube-double-play\texttt{cube-double-play}11.1689.631.7
ManiSkill3 MPStackCube\texttt{StackCube}1.37*32.852.3
ManiSkill3 PPOStackCube\texttt{StackCube}5.8369.929.5
RoboTwin 2.0\texttt{stack_blocks_two}1.46*33.732.0
RLBench\texttt{stack_blocks}3.00*35.159.4
Robomimic MGlift\texttt{lift}19.0468.728.2

The table above compares the three metrics of OCBench, two human-generated datasets and six previous scripted or policy-generated datasets. The results support that OCBench has more similar properties than other synthetic datasets in general. While these metrics aren't perfect (they are proxies), we believe these complementary metrics, along with our video comparison, support our claim about structural similarity.

(More details: For action saturation, we measure the fraction of actions that have ∣a∣≥0.95|a| \geq 0.95 after normalization, where asterisks mean we measure them after converting to joint deltas. For speed consistency, we measure the ratio of speed at bends to speed on nearby portions (within 2.5cm) of the trajectory. For turning concentration, we measure the fraction of directional change concentrated in the (sharpest) 10% of path locations within 10cm windows. Also, in this comparison, we focus on purely synthetic datasets, excluding MimicGen-generated ones (e.g., MimicGen and RoboCasa), whose datasets are generated by augmenting ~10 human trajectories; see our discussion (L789).)

  • "Do you have any examples of phenomena or hypotheses which only OCBench is capable of demonstrating or rejecting, while every other dataset fails at doing these?"

Thanks for the question. We'd like to first clarify the scope of our claim. We didn't intend to claim that these phenomena are reproducible only in OCBench. That said, we do show that previous synthetic benchmarks (OGBench and Robomimic MG) do not show a substantial gap between open-loop and closed-loop policies (Phenomenon 2; see Figure 15 in Appendix) and require only small models (Phenomenon 3; see Figure 16 in Appendix). Such gaps make it difficult to study these phenomena in the previous synthetic benchmarks. Moreover, to our knowledge, OCBench is the first documented reproduction of the dataset size-reversal phenomenon (Figure 4) in a robotic benchmark with scripted policies.

To further demonstrate that previous synthetic datasets do not always reproduce all the phenomena discussed in the paper, we perform additional experiments with three additional scripted or policy-generated datasets, MetaWorld (scripted), ManiSkill3 (policy-generated), and RoboTwin (scripted), on similar block manipulation tasks.

DatasetTaskClosed-loopOpen-loop
OCBench (ours)block-single-task1\texttt{block-single-task1}8±58 \pm 582±382 \pm 3 (K=25K=25)
OGBenchcube-single-task2\texttt{cube-single-task2}94±894 \pm 8100±0100 \pm 0 (K=5K=5)
Robomimic MGlift\texttt{lift}34±834 \pm 825±425 \pm 4 (K=5K=5)
MetaWorldpick-place\texttt{pick-place}100±0100 \pm 0100±0100 \pm 0 (K=5K=5)
ManiSkill3StackCube\texttt{StackCube}82±682 \pm 666±066 \pm 0 (K=5K=5)
RoboTwin 2.0\texttt{stack_blocks_two}0±00 \pm 060±360 \pm 3 (K=25K=25)
DatasetTaskWidth 512Width 1024Width 2048Width 4096Width 8192
OCBench (ours)block-double-task1\texttt{block-double-task1}0±00 \pm 029±329 \pm 354±254 \pm 267±167 \pm 171±571 \pm 5
OGBenchcube-single-task2\texttt{cube-single-task2}(saturated) 98±098 \pm 0100±0100 \pm 0100±0100 \pm 0——
Robomimic MGlift\texttt{lift}(saturated) 46±646 \pm 635±135 \pm 140±340 \pm 3——
MetaWorldpick-place\texttt{pick-place}(saturated) 100±0100 \pm 0100±0100 \pm 0100±0100 \pm 0——
ManiSkill3StackCube\texttt{StackCube}(saturated) 82±682 \pm 680±880 \pm 877±177 \pm 1——
RoboTwin 2.0\texttt{stack_blocks_two}60±360 \pm 3(saturated) 97±197 \pm 199±199 \pm 1——

The first table compares the performance of closed-loop and open-loop policies (Phenomenon 2; KK means the action chunk length, where we chose the best KK from 5,25\\{5, 25\\} for each dataset), and the second table shows the model size ablation (Phenomenon 3; "saturated" means little or no further improvement over the larger widths). The first table shows that 4 out of 5 previous synthetic benchmarks show little or no improvement from action chunking (or closed-loop control is already enough). While RoboTwin 2.0 does reproduce this phenomenon, its performance saturates at width 1024 (the second table), whereas OCBench and human demonstrations (Pan et al., 2026) benefit from much larger models, which we hypothesize is due to their behavioral complexity. We also note that RoboTwin 2.0 also exhibits qualitatively more mechanistic behavior than OCBench, as shown in our video comparison.

These results suggest that previous synthetic datasets may not consistently reproduce all the phenomena discussed in the paper. While individual phenomena may still partially be reproducible in other benchmarks, OCBench is, to our knowledge, the first attempt to compile and systematically reproduce these phenomena together in a controlled benchmark, with scripted policies that have more similar structural properties to human demonstrations. OCBench provides a convenient playground for studying BC (and potentially RL), which we believe is valuable to the community.


We would like to thank you again for raising important questions about our work. We believe the additional results and clarifications have strengthened the paper. Please let us know if you have any additional concerns or questions. If we have addressed your concerns, we would appreciate it if you could adjust the score accordingly.

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer BLuy,

Thank you again for the thoughtful review and feedback. Have the additional quantitative metrics and new comparisons with additional previous benchmarks above mostly addressed your concerns? Please feel free to let us know if you have any other remaining questions or concerns.

Thanks,
Authors

Reviewer_iDStOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to systematically investigate empirical phenomena in behavioral cloning. The paper proposes OCBench, which includes scripted policies that can automatically generate demonstrations with structural properties similar to human demonstrations, using Hermite splines and randomized data-generation parameters. The authors then conduct controlled studies using OCBench. The experiments reproduce several interesting phenomena in behavioral cloning, some of which are not obvious in previous benchmarks. The paper also proposes hypotheses to explain these phenomena and provides insights for future research. Overall, I think this is an interesting and valuable paper that should be accepted.

Critical strengths
  1. OCBench can generate demonstrations with human-like properties, which is interesting and useful for studying behavioral cloning.
  2. The analyses of the phenomena are carefully conducted and provide valuable insights for future research.
Critical weaknesses
  1. The scripted data-generation policy is a main contribution of the paper, but it is not sufficiently explained. It is unclear how the control points are generated, how mistakes are introduced and recovered from, what distributions are used for the randomized parameters, and how these distributions are determined.
  2. It is unclear which components of the proposed scripted policies are most important for generating human-like behaviors. The smooth Hermite splines, the randomization of parameters, or both? Besides, the comparison between OCBench and previous benchmarks is not conducted on the same tasks and environments. Appendix C mentions that the authors ran OGBench's scripted policy in an OCBench environment for Figure 1, but did not train BC policies on these demonstrations. For a more controlled comparison, the authors can generate demonstrations using both OGBench's scripted policy and OCBench's proposed scripted policy on the same OCBench task, train the same BC architectures on both datasets, and evaluate whether the four phenomena emerge under each data-generation strategy.
Questions for authors
  1. Do the observed phenomena depend on the action representation? The paper focuses on delta joint actions. Would the same phenomena be observed with end-effector pose or absolute actions? Would the differences between OCBench and previous benchmarks hold under different action spaces?
  2. In Figure 18 (c) the pixel-based policy outperforms the state-based policy and achieves nearly perfect success rate, which is somewhat counterintuitive. Do the authors have an explanation for this?
2 comments

Thank you for the detailed review and constructive feedback on this work. We especially appreciate your comments about action representations and OCBench vs. OGBench comparison. Following your suggestion, we have conducted three additional experiments with absolute joint position control and OGBench on OCBench tasks and clarified how the scripted policies work. Please find our response below.

  • "Do the observed phenomena depend on the action representation?", "Would the differences between OCBench and previous benchmarks hold under different action spaces?"

Thanks for the question. Following the suggestion, we reproduced the main experiments in Phenomena 1 and 2 using absolute joint position-based control.

Training updatesTraining loss (×10−3\times 10^{-3})Validation loss (×10−3\times 10^{-3})Success (%)
200K3.31±0.533.31 \pm 0.533.32±0.083.32 \pm 0.0811±711 \pm 7
500K2.72±0.602.72 \pm 0.603.35±0.613.35 \pm 0.6131±1231 \pm 12
1M1.64±0.061.64 \pm 0.064.57±0.764.57 \pm 0.7643±843 \pm 8
2M0.72±0.180.72 \pm 0.184.85±0.504.85 \pm 0.5057±2057 \pm 20
TaskClosed-loopOpen-loop
block-single-task1\texttt{block-single-task1}0±00 \pm 051±351 \pm 3 (K=25K=25)
hanoi-single-task1\texttt{hanoi-single-task1}0±00 \pm 040±1640 \pm 16 (K=25K=25)

The first table shows training "curves" on block-single-task1\texttt{block-single-task1} and the second table compares closed-loop and open-loop control (KK means the action chunk length). These tables respectively show that overfitting is not necessarily harmful and open-loop control substantially outperforms closed-loop control, in joint position-based control too. We'd like to note that previous works have also reported similar phenomena from real human demonstrations under joint position-based control (e.g., the original ACT paper by Zhao et al. (2023)), suggesting that these phenomena are not limited to a single type of action representation.

Next, we show whether the difference between OCBench and a previous benchmark (OGBench) holds under absolute joint position control too.

Scripted policyTaskClosed-loopOpen-loop
OCBench policy (ours)block-single-task1\texttt{block-single-task1}0±00 \pm 051±351 \pm 3 (K=25K=25)
OGBench policyblock-single-task1\texttt{block-single-task1}87±287 \pm 296±196 \pm 1 (K=10K=10)

The table above compares closed-loop and open-loop policies learned from OCBench and OGBench scripted policies. We used the same OCBench tasks in this comparison (as the reviewer suggested; see our response to the next question too). The results show that the policy learned from OGBench-style demonstrations already achieves 87% with closed-loop control, whereas the policy learned from OCBench-style demonstrations achieves zero success without action chunking. Moreover, the former (OGBench) uses a much smaller 512-sized network, as opposed to the 4096-sized network used for OCBench-style demonstrations. These results suggest that the gap between OCBench and OGBench (OCBench's closest predecessor) is not limited to the specific action representation used in the paper.

  • "The comparison between OCBench and previous benchmarks is not conducted on the same tasks and environments."

Thanks for the suggestion. We performed an apples-to-apples comparison between OGBench- and OCBench-style demonstrators on OCBench's basic block manipulation tasks. We used the same environment and action interface (i.e., 7-D joint delta control), while only varying scripted policy-specific designs, following the suggestion.

Scripted policyTaskClosed-loopOpen-loop
OCBench policy (ours)block-single-task1\texttt{block-single-task1}8±58 \pm 582±382 \pm 3 (K=25K=25)
OGBench policyblock-single-task1\texttt{block-single-task1}96±196 \pm 196±196 \pm 1 (K=10K=10)
Scripted policyTaskWidth 512Width 4096Width 8192
OCBench policy (ours)block-double-task1\texttt{block-double-task1}0±00 \pm 067±167 \pm 171±571 \pm 5
OGBench policyblock-double-task1\texttt{block-double-task1}95±295 \pm 297±197 \pm 1—

The first table compares closed- and open-loop control and the second table compares different model sizes. The results show that OGBench's scripted policy benefits less from action chunking, and requires much smaller models, unlike OCBench and human demonstrations, matching the original result in the paper.

  • "The scripted data-generation policy is a main contribution of the paper, but it is not sufficiently explained."

Thanks for the suggestion! Following the suggestion, we have updated the paper to include more details about the scripted policies in Appendix B (marked in red).

  • "In Figure 18 (c) the pixel-based policy outperforms the state-based policy and achieves nearly perfect success rate, which is somewhat counterintuitive. Do the authors have an explanation for this?"

Great point! We also found this to be a bit counterintuitive, but related observations have been reported in previous papers with real-world human demonstrations: Lu et al. (2026) show that vision-only policies often outperform vision-plus-proprioception policies, and the Octo paper also made a similar observation in Appendix E. We hypothesize that this is due to the test-time distribution shift issue. The plots in Figure 18 (c) show that pixel-based BC achieves lower test-time MSE even though it has higher validation MSE (just like the analyses in Phenomenon 4), suggesting that pixel-based policies might generalize better at test time (potentially because OOD values in state-based observations might have more harmful effect).


We would like to thank you again for raising important questions about our work. We believe the additional results and clarifications have strengthened the paper. Please let us know if you have any additional concerns or questions. If we have addressed your concerns, we would appreciate it if you could adjust the score accordingly.

Reviewer_MqqhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OCBench is a GPU-accelerated (MJWarp) manipulation benchmark for studying behavioral cloning (BC).

  • Tasks: 28 state-based tasks across five environment families, plus 10 Hz "lite" and pixel-based variants. The robot is a UR5e with 7-D joint-delta actions at 50 Hz.
  • Demonstrators: scripted policies built from randomized C1-Hermite splines, designed to be "human-like": narrowly distributed, non-Markovian and temporally correlated, semantically diverse, and occasionally making and recovering from mistakes.
  • Speed: data generation reaches up to 200K environment steps/s on one H100, which makes streaming ("unlimited data") training practical.

The paper claims two contributions:

  1. the benchmark and its human-like scripted policies;
  2. controlled reproductions of four previously reported BC phenomena, with tests of candidate hypotheses for each:
    • overfitting is not harmful, including a reversal where a larger dataset does worse;
    • action chunking far outperforms closed-loop control, and history conditioning hurts;
    • policies need to be very large (up to ~0.5B parameters on state-based tasks);
    • input features matter even with unlimited data.

Validation:

  • Flow-matching BC with 25-step action chunks, mostly with 2 seeds and on 1–2 tasks per phenomenon.
  • Appendix comparisons against OGBench and Robomimic-MG (Figs. 15–16).
  • Per-dimension action histograms against a human dataset (Fig. 1).
  • Preliminary pixel-based results.
  • Reference BC / flow-BC / FSQ-BC results on all tasks.
Critical strengths
  1. Fast, controllable infrastructure. Demonstrators can be regenerated on demand, with subtask labels, full simulator state, and knobs for speed and mistake rate. This supports experiments that fixed human datasets cannot. Streaming-data training in particular separates data scarcity from algorithmic effects. This is the paper's main value, and the reason I see OCBench as a potentially useful community resource.

  2. Two controlled results that go beyond earlier anecdotal reports.

    • Closed-loop policies still collapse with unlimited streaming data (Fig. 6).
    • The dataset-size reversal also appears with i.i.d. resampled datasets (Fig. 4). This shows it can arise without the distribution mismatch suspected by Codevilla et al. (2019) and Bai et al. (2025).

    Both results need a controllable data generator. They are good examples of the studies the benchmark is meant to enable.

Critical weaknesses

1. The central premise is assumed rather than tested. The paper's premise is that OCBench reproduces these phenomena because its demonstrations are human-like. That premise is never tested directly, and it is entangled with system and learner choices that are held fixed.

  • Human-likeness is not established.: It is defined through a set of hypothesized properties, but neither OCBench's data nor human data is shown to have them in a measurable way -- also the comparison with human data is qualitative. The successful reproduction of the phenomena is itself offered as evidence of human-likeness, which is kind of an circular argument..

  • The generator's knobs are barely used to test it.: They could test the premise directly, by moving the data toward or away from these properties in the same environment. However, this is done only once, by removing temporal correlation, and that experiment only partially supports the claim.

  • The comparison with earlier synthetic benchmarks cannot isolate the data. The robot, action space, control rate, and task difficulty all change along with the demonstrator.

  • Every experiment fixes the same system and learner. That means one control rate, one action parameterization and low-level controller, and essentially one policy class.

    • These choices are known to shape what policies can learn and which methods look best (e.g., Varin et al., 2019; Martín-Martín et al., 2019; Aljalbout et al., 2024; Bronars et al., 2026). For BC specifically, Chi et al. (2025) compare position and velocity control.
    • Recent analysis of action chunking (Lazzati et al., 2026) also suggests that its benefit does not simply track how non-Markovian the demonstrator is.
    • So it remains open whether the reproduced phenomena are properties of BC from human-like data, or of this particular configuration.
  • Why this matters. Much of the analysis revisits known explanations and diagnostics (see Question 4), so the contribution rests largely on OCBench being a faithful proxy for BC on real demonstrations. A benchmark that bakes in a single configuration risks presenting configuration-specific behavior as general BC behavior. A benchmark meant for "scientific studies" of BC should guard against exactly that.

2. The empirical support for each phenomenon is narrow.

  • Each phenomenon is demonstrated on a small number of tasks, with few seeds.
  • Some effects, notably the dataset-size reversal, are small and could plausibly be explained by the training budget chosen.

For a benchmark whose purpose is to make these phenomena reliably studyable, I would expect evidence that they hold across a broader set of its own tasks.

Questions for authors

1. What exactly makes the data "human-like," and is that what drives the phenomena? (Clarification + experiment)

Could the authors:

  • quantify the proposed properties for OCBench, prior synthetic data, and human data;
  • show how the phenomena change as these properties are varied within the same environment and action interface? This could use the generator's own knobs, or an OGBench-style demonstrator run in OCBench.

Even a small set of teleoperated demonstrations on one task would help. Clear evidence here would substantially raise my assessment.

2. Statistics. (Clarification)

  • How is uncertainty estimated, given the small number of seeds?
  • Does the dataset-size reversal persist when the two dataset sizes are compared at matched training epochs, or when both are trained to convergence?

References not in the paper's bibliography:

  • Aljalbout et al. On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer. RA-L 2024.
  • Bronars et al. Tune to Learn: How Controller Gains Shape Robot Policy Learning. RSS 2026.
  • He et al. Demystifying Robot Diffusion Policies: Action Memorization and a Simple Lookup Table Alternative. ICLR 2026.
  • Liu et al. Bidirectional Decoding: Improving Action Chunking via Closed-Loop Resampling. arXiv:2408.17355.
  • Martín-Martín et al. Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks. IROS 2019.
  • Nakkiran et al. Deep Double Descent: Where Bigger Models and More Data Hurt. ICLR 2020.
  • Varin et al. A Comparison of Action Spaces for Learning Manipulation Tasks. IROS 2019.
8 comments

Thank you for the detailed review and constructive feedback on this work. We especially appreciate your comments about human-likeness as well as other valid questions about our empirical results. Following your suggestion, we have conducted three additional experiments, measured three quantitative metrics for dataset trajectories, and performed statistical analyses. Please find our response below.

  • "Human-likeness is not established" / Could the authors "quantify the proposed properties for OCBench, prior synthetic data, and human data"?

Thanks for raising this point. We'd like to first clarify that, by "human-like", we meant OCBench trajectories have similar structural properties to human demonstrations (e.g., temporally correlated, smooth, etc.). We didn't intend to claim that the synthetic trajectories themselves are necessarily very similar to human demonstrations, which is a much more subjective and harder-to-justify claim. While we tried to convey this point throughout the paper (e.g., L44, Sections 2-3), we found that some phrases might potentially be interpreted in the other way, and we've adjusted wording (changes in red) and further clarified this point in L139.

Then, regarding whether OCBench really has more similar structural properties to human demonstrations than previous scripted policies, we would like to share two perspectives.

First, following the suggestion, we measured three quantitative metrics, (1) action saturation (the one used in the paper), (2) speed consistency (measuring how much trajectories are halting/phase-y), and (3) turning concentration (how "sharp" turning movements are), hypothesizing that they capture some structural properties (e.g., smoothness) of human demonstrations. We measured them on datasets from OCBench, previous synthetic benchmarks, and human demonstrations.

(More details: For action saturation, we measure the fraction of actions that have ∣a∣≥0.95|a| \geq 0.95 after normalization, where asterisks mean we measure them after converting to joint deltas. For speed consistency, we measure the ratio of speed at bends to speed on nearby portions (within 2.5cm) of the trajectory. For turning concentration, we measure the fraction of directional change concentrated in the (sharpest) 10% of path locations within 10cm windows.)

DatasetTaskAction saturation (%)Speed consistency (%)Turning concentration (%)
Human demo (Robomimic PH)lift\texttt{lift}1.6767.231.9
Human demo (BiPlay)play\texttt{play}1.13*73.825.0
OCBench (ours)block-double-task1\texttt{block-double-task1}0.2278.631.8
OGBenchcube-double-play\texttt{cube-double-play}11.1689.631.7
ManiSkill3 MPStackCube\texttt{StackCube}1.37*32.852.3
ManiSkill3 PPOStackCube\texttt{StackCube}5.8369.929.5
RoboTwin 2.0\texttt{stack_blocks_two}1.46*33.732.0
RLBench\texttt{stack_blocks}3.00*35.159.4
Robomimic MGlift\texttt{lift}19.0468.728.2

The table above compares the three metrics of OCBench, two human-generated datasets and six previous scripted or policy-generated datasets. The results support that OCBench has more similar properties than other synthetic datasets in general. While these metrics aren't perfect (they are proxies), we believe these metrics (partly) support our claim about structural similarity.

Second, we also provide a complementary video comparison between OCBench, human demonstrations, and previous synthetic datasets at https://ocbench-videos.pages.dev/. The videos show that OCBench trajectories are much smoother, less "phase-y"/mechanistic, and more diverse (e.g., random tilts, retries, etc.) than the previous ones (even though OCBench trajectories may not be perfectly "human-like"). While this is still a qualitative comparison, it provides a complementary perspective on why OCBench and previous benchmarks may have different metrics in the above table.

  • Could the authors "show how the phenomena change as these properties are varied within the same environment and action interface? This could use the generator's own knobs, or an OGBench-style demonstrator run in OCBench"

Thanks for the suggestion. We performed an apples-to-apples comparison between OGBench- and OCBench-style demonstrators on OCBench's basic block manipulation tasks. We used the same environment and action interface (i.e., 7-D joint delta control), while only varying scripted policy-specific designs, following the suggestion.

Scripted policyTaskClosed-loopOpen-loop
OCBench policy (ours)block-single-task1\texttt{block-single-task1}8±58 \pm 582±382 \pm 3 (K=25K=25)
OGBench policyblock-single-task1\texttt{block-single-task1}96±196 \pm 196±196 \pm 1 (K=10K=10)
Scripted policyTaskWidth 512Width 4096Width 8192
OCBench policy (ours)block-double-task1\texttt{block-double-task1}0±00 \pm 067±167 \pm 171±571 \pm 5
OGBench policyblock-double-task1\texttt{block-double-task1}95±295 \pm 297±197 \pm 1—

The first table compares closed- and open-loop control (KK means the action chunk length) and the second table compares different model sizes. The results show that OGBench's scripted policy benefits less from action chunking, and requires much smaller models, unlike OCBench and human demonstrations, matching the original results in the paper.

  • "How is uncertainty estimated, given the small number of seeds?"

While we used two seeds for most experiments in this paper (due to the high computational cost of training 100M-scale models under a limited compute budget), we found that the performance gaps are mostly clear enough to support our conclusions, and we added more seeds otherwise.

To quantitatively support this claim, we have performed two-sided Welch’s t-tests for the main experiments for each of the four phenomena.

FigureTaskComparisonFinal success rate# seedsp-value
Figure 4block-single\texttt{block-single}10K vs. 50K episodes90.90% vs. 85.45%40.0429
Figure 4chamber-easy\texttt{chamber-easy}10K vs. 30K episodes51.70% vs. 43.80%40.0448
Figure 6block-single\texttt{block-single}Open-loop vs. closed-loop81.80% vs. 7.80%20.0085
Figure 6hanoi-single\texttt{hanoi-single}Open-loop vs. closed-loop70.10% vs. 0.50%20.0045
Figure 10block-double\texttt{block-double}Width 4096 vs. 102466.80% vs. 29.30%20.0339
Figure 10hanoi-single\texttt{hanoi-single}Width 4096 vs. 102470.40% vs. 48.50%20.0736
Figure 12block-double\texttt{block-double}Handcrafted vs. standardized + xyz×0.1xyz \times 0.164.30% vs. 2.00%20.0089
Figure 12hanoi-single\texttt{hanoi-single}Handcrafted vs. standardized + xyz×0.1xyz \times 0.172.30% vs. 55.60%20.0620

The table above shows the statistical test results, which suggest that the gaps are mostly statistically meaningful despite the small number of seeds.

  • "Does the dataset-size reversal persist when the two dataset sizes are compared at matched training epochs, or when both are trained to convergence?"

Figure 4 compares performance under the same training budget (with identical batch sizes and optimization settings). Note also that both datasets are sampled from the same trajectory distribution. Although performance improvements slowed over training, they did not fully converge within our computational budget, so additional training might eventually close the gap. Nevertheless, we believe the performance reversal under the same training budget is itself a noteworthy observation. We have clarified this in the paper (L215).

  • "Every experiment fixes the same system and learner." Does position-based control change the results?

Thanks for the question. Following the suggestion, we reproduced main experiments in Phenomena 1 and 2 under absolute joint position-based control.

Training updatesTraining loss (×10−3\times 10^{-3})Validation loss (×10−3\times 10^{-3})Success (%)
200K3.31±0.533.31 \pm 0.533.32±0.083.32 \pm 0.0811±711 \pm 7
500K2.72±0.602.72 \pm 0.603.35±0.613.35 \pm 0.6131±1231 \pm 12
1M1.64±0.061.64 \pm 0.064.57±0.764.57 \pm 0.7643±843 \pm 8
2M0.72±0.180.72 \pm 0.184.85±0.504.85 \pm 0.5057±2057 \pm 20
TaskClosed-loopOpen-loop
block-single-task1\texttt{block-single-task1}0±00 \pm 051±351 \pm 3 (K=25K=25)
hanoi-single-task1\texttt{hanoi-single-task1}0±00 \pm 040±1640 \pm 16 (K=25K=25)

The first table shows training "curves" on block-single-task1\texttt{block-single-task1} and the second table compares closed-loop and open-loop control. These tables respectively show that overfitting is not necessarily harmful and open-loop control substantially outperforms closed-loop control, in joint position-based control too. We'd like to note that previous works have also reported similar phenomena from real human demonstrations under joint position-based control (e.g., the original ACT paper by Zhao et al. (2023)), suggesting that these phenomena are not limited to a single type of action representation.

  • "Each phenomenon is demonstrated on a small number of tasks"

In the paper, we focused on a few representative tasks to demonstrate each phenomenon due to our compute constraints (note that each 4096-sized model has ~100M parameters). However, these phenomena are not limited to those tasks.

For example, Phenomenon 1 appears broadly across many tasks in OCBench. The table below shows the training loss (×10−3\times 10^{-3}), validation loss (×10−3\times 10^{-3}), and performance trends on 10 tasks from our benchmark table (Table 6). Each entry shows flow BC results at 500K → 1M → 2M training updates.

TaskTraining lossValidation lossSuccess (%)
block-double-task2\texttt{block-double-task2}13.65→10.53→4.3013.65 \to 10.53 \to 4.3030.36→29.95→48.8830.36 \to 29.95 \to 48.8840±7→59±8→72±540 \pm 7 \to 59 \pm 8 \to 72 \pm 5
block-quadruple-task3\texttt{block-quadruple-task3}16.53→14.03→10.9716.53 \to 14.03 \to 10.9719.95→19.16→21.9719.95 \to 19.16 \to 21.970±0→1±1→7±20 \pm 0 \to 1 \pm 1 \to 7 \pm 2
chamber-easy-task1\texttt{chamber-easy-task1}11.71→5.54→3.5511.71 \to 5.54 \to 3.5520.85→29.77→41.6020.85 \to 29.77 \to 41.6064±4→70±12→73±1064 \pm 4 \to 70 \pm 12 \to 73 \pm 10
chamber-easy-task3\texttt{chamber-easy-task3}10.81→5.32→4.5110.81 \to 5.32 \to 4.5132.48→49.79→67.5432.48 \to 49.79 \to 67.5431±0→39±9→48±1031 \pm 0 \to 39 \pm 9 \to 48 \pm 10
chamber-medium-task1\texttt{chamber-medium-task1}17.83→12.83→7.3917.83 \to 12.83 \to 7.3924.59→25.58→33.0024.59 \to 25.58 \to 33.002±1→7±1→11±12 \pm 1 \to 7 \pm 1 \to 11 \pm 1
chamber-medium-task2\texttt{chamber-medium-task2}14.71→12.04→6.2014.71 \to 12.04 \to 6.2020.43→26.90→42.8620.43 \to 26.90 \to 42.860±0→3±1→5±20 \pm 0 \to 3 \pm 1 \to 5 \pm 2
hanoi-single-task2\texttt{hanoi-single-task2}5.27→3.01→1.875.27 \to 3.01 \to 1.8718.90→26.28→33.9518.90 \to 26.28 \to 33.9572±4→72±6→73±372 \pm 4 \to 72 \pm 6 \to 73 \pm 3
hanoi-double-task2\texttt{hanoi-double-task2}11.16→7.19→5.9311.16 \to 7.19 \to 5.9310.66→8.84→11.6710.66 \to 8.84 \to 11.675±0→12±3→17±125 \pm 0 \to 12 \pm 3 \to 17 \pm 12
switch-3x3-task1\texttt{switch-3x3-task1}4.03→2.20→2.004.03 \to 2.20 \to 2.00136.81→172.71→191.13136.81 \to 172.71 \to 191.1310±0→15±1→16±110 \pm 0 \to 15 \pm 1 \to 16 \pm 1
switch-3x3-task2\texttt{switch-3x3-task2}4.46→2.08→1.924.46 \to 2.08 \to 1.9296.49→104.86→152.2396.49 \to 104.86 \to 152.2311±1→14±1→17±011 \pm 1 \to 14 \pm 1 \to 17 \pm 0

We also performed open- vs. closed-loop ablation on two additional tasks (Phenomenon 2):

TaskClosed-loopOpen-loop
block-double-task2\texttt{block-double-task2}0±00 \pm 072±572 \pm 5 (K=25K=25)
chamber-easy-task1\texttt{chamber-easy-task1}0±00 \pm 073±1073 \pm 10 (K=25K=25)

These results show that the phenomena studied in the paper are not limited to the tasks analyzed in the paper. Indeed, one of the main goals of this paper is to reproduce non-intuitive phenomena that researchers have previously observed with real-world human demonstrations (Section 3) in a controlled setting.


We would like to thank you again for raising important questions about our work. We believe the additional results and clarifications have strengthened the paper. Please let us know if you have any additional concerns or questions. If we have addressed your concerns, we would appreciate it if you could adjust the score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.