- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks when offline imitation error is a good proxy for picking a controller, using one real PTZ camera tracking one person, seven BC variants and three rounds of a single DAgger chain. Every DAgger checkpoint beats every BC checkpoint on held-out error, while in-distribution error favours BC, and Eq. 3 splits this reversal into an in-distribution margin and a shift penalty. BC trained on the demonstrations plus the DAgger-collected states recovers most of DAgger's gain, and removing the RGB input cuts held-out error by an order of magnitude. A pre-registered live test then shows that the in-distribution choice wins at picking the best policy, while held-out error ranks the ten policies much better.
Critical strengths
-
The hardware protocol is very careful. Checkpoints are deployed byte-identical in counterbalanced blocks, the primary endpoint was locked and hashed before data collection, and exploratory results are clearly labelled. Since the claims are about rankings flipping, this is what makes me trust the results.
-
The paper separates things that usually get lumped together: the effect of the collected states versus the warm start, with matched budgets, and ranking policies versus picking the best one. The live test makes a point I hadn't seen put this clearly: a proxy can rank policies well and still pick a worse deployed policy.
Critical weaknesses
-
The headline result comes from one DAgger collection chain, and there's a plausible alternative explanation. The seeds only re-run fine-tuning on the same collected data (App. H), the margin between the two families is narrow next to BC seed variance, and the in-distribution reversal shows up on only one of the two validation clips. DAgger also roughly doubles the training data, and RQ3 shows that RGB drives most of the shift penalty: geometry-only BC on the demonstrations alone is about 4x better than the best DAgger checkpoint. So RQ1 may be more about giving an RGB policy more varied frames than about the collection regime, and once stronger candidates are added, the offline correlation turns positive (App. M).
-
The setting is narrow and the link to deployment is mixed. There's one camera, one training actor, two held-out clips and one live setting. Under free zoom, the pre-registered primary endpoint favours the in-distribution choice, neither proxy ranks coverage well, and the 2x-floor sessions that favour the held-out choice are exploratory. The lesson in Sec. 8 is sensible but close to standard advice.
Questions for authors
- Is a second DAgger collection chain feasible? Without one there's no way to estimate collection-level uncertainty for RQ1.
- Could you add BC trained on the same number of extra frames collected independently of the learner? Sec. 5 names this control as missing.
- Does the inversion still hold if all ten candidates are geometry-only or use a frozen encoder? If not, I'd scope RQ1 to candidate sets with an appearance nuisance.
- Could you add statistics for the held-out clips to Table 5, and the held-out ranking with each clip dropped?
- Training longer on the demonstrations alone already lowers flat BC's held-out error a lot (App. L). Were the other BC candidates trained to convergence?
- Is participant video released, and does the consent cover that? The Ethics Statement also notes that no formal ethics-board review was done.
Other suggestions:
- Sec. 7 says FOV-risk beats flat BC in all five rounds, but Table 11 shows a tie in round 2 (0.26 vs 0.26). Unrounded values would settle it.
- Appendix G appears as an empty heading because Table 8 floats away from it.
- Possibly relevant: Gulrajani and Lopez-Paz (ICLR 2021) on domain generalization, Miller et al. (ICML 2021) on accuracy on the line, Irpan et al. (NeurIPS 2019) on off-policy classification, and Wen et al. (NeurIPS 2020) on copycat agents in behavioral cloning.
Round 2 follow-up: unseen people, resampling sensitivity and matched-row data
Follow-ups with the revised PDF.
One training person (critical weakness 2). We ran a post-review test on two previously unseen people, plus the original person in new clothing as a same-person control (general comment, new Appendix S). This adds external test people; training still used one person. Against BC trained for 60 epochs, DAgger-R2 had lower live J_0.25 for both new people: -0.17 (95% CI [-0.25, -0.08]) and -0.43 ([-0.50, -0.36]). Its offline error on their clips was also higher (0.123 vs. 0.117 and 0.235 vs. 0.044). The control showed the same reversal, so the result limits DAgger-over-BC across training budgets and cannot be attributed solely to testing unseen people. It is consistent with our round-1 answer to Q5: the paper's held-out error already ranked this BC checkpoint above DAgger-R2. At the 20-epoch budget of Table 2, DAgger-R2 beat L=1 and flat BC in 6/6 blocks for every person. We keep the limitation: one training person, one room per phase, one rig and one DAgger chain.
Q1, a second DAgger chain. A new chain is still not feasible. Appendix R.2 reports a conditional resampling sensitivity analysis, fixed before any training. It is not a second collection chain. Blocks of 300 frames from the collected data were resampled within each round, and the R1->R2->R3 fine-tuning chain was retrained 100 times. Within the original 20-epoch comparator set, DAgger-R2 had lower held-out error than all 21 BC-family checkpoints in 100/100 replicates. Its gap to the best of them has a 95% interval of [0.014, 0.092] (median 0.061). The full cross-family inversion held in 84/100 replicates, so it does not meet our pre-stated 90% robustness criterion. All 16 failures are R1 or R3 reaching the best BC checkpoint. Every resampled segment still comes from the one original collection trajectory and its policies, so the limitation stays in the paper.
Q2, data doubling. We also matched total rows (Appendix R.1, protocol fixed before training). Every arm adds 1,685 rows to the same base set (1,216 with a do command) and runs 8,568 updates. Held-out error is still lower with collected rows than with clip6 demonstration frames. The medians are 0.064 for round-3 rows and 0.045 for rounds 1-2, against 0.147, and each is better in 5/5 paired seeds. In-distribution error on the validation clip is higher (0.0010 and 0.0014 vs. 0.0005), the same inversion as in RQ1. Under these conditions, row count alone does not explain the held-out difference. These results do not identify which property of the collected data causes it (collecting policy, state distribution or corrections). The original DAgger-R2 held-out advantage over the compared 20-epoch BC checkpoints does not extend to the 60-epoch L=1 checkpoint, which already had lower held-out error. Post hoc, with the rule locked before computing (Appendix S), the collected-data arm trained from scratch at the 60-epoch budget (five seeds) had lower error on all three newly recorded clips (P0 control, P1 and P2) than the best 60-epoch BC recipe: medians 0.031, 0.022 and 0.057 vs. 0.102, 0.087 and 0.062. It was not deployed live and is not the chain recipe, so it does not show that DAgger beats 60-epoch BC in closed loop. In short, the new evidence does not supply multi-person training, an independent DAgger chain or a causal account of the collected data.