acceptodds
Under review as a conference paper at ICLR 2027

Offline Imitation Error as a Model-Selection Proxy Under Distribution Shift: Evidence from Real-Camera Active PTZ Control

Abstract

Practitioners often screen imitation-learning policies with logged-state imitation error - per-step action error against an expert on logged data - before expensive closed-loop evaluation. We study when this proxy is valid on a real pan-tilt-zoom (PTZ) camera for indoor single-target tracking. (RQ1) Does in-distribution error preserve the ordering of held-out imitation error? Not across data-collection regimes: across three optimisation runs the held-out BC-DAgger separation persists, while in-distribution error favours behaviour-cloning (BC) policies in 178/189 pooled cross-family comparisons; the DAgger reruns share one fixed collection chain, and orderings within the BC family do not survive retraining. (RQ2) Controlled retraining shows that access to the DAgger-collected support is sufficient for most of the held-out gain, and the warm-start fine-tuning trajectory is not required to recover it: plain BC from scratch on the collected states recovers it under matched-epoch and approximately matched-update budgets, without DAgger's in-distribution degradation; we do not show that learner-induced states are uniquely responsible. (RQ3) For our geometry-defined expert, a pretrained RGB branch raises held-out error 10-17x even though the sufficient geometric inputs are given explicitly; in separate controls, constant and shuffled images stay much closer to geometry-only, consistent with the cost depending on state-aligned visual content rather than on the presence or size of the image branch, and a protocol-locked external check agrees in direction but is imprecise. Finally, a pre-specified live test of the same ten checkpoints separates selection from ranking: under free zoom the in-distribution choice has the highest mean telemetry-derived centred-tracking score and more in-frame coverage than the held-out choice in 5/5 blocks, while held-out error orders centred tracking across all ten far better (Spearman +0.71 vs. +0.01). Offline imitation error is thus not one model-selection signal: its validity depends on where it is evaluated, how the candidates acquired their training support, and whether the goal is to select one policy or rank many. All physical-system results come from one PTZ rig.

Primary area:
applications to robotics, autonomy, planning
Keywords:
imitation learning, model selection, offline policy selection, distribution shift, behavior cloning, DAgger, closed-loop evaluation, active perception, PTZ camera control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QZDqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when offline imitation error is a good proxy for picking a controller, using one real PTZ camera tracking one person, seven BC variants and three rounds of a single DAgger chain. Every DAgger checkpoint beats every BC checkpoint on held-out error, while in-distribution error favours BC, and Eq. 3 splits this reversal into an in-distribution margin and a shift penalty. BC trained on the demonstrations plus the DAgger-collected states recovers most of DAgger's gain, and removing the RGB input cuts held-out error by an order of magnitude. A pre-registered live test then shows that the in-distribution choice wins at picking the best policy, while held-out error ranks the ten policies much better.

Critical strengths
  1. The hardware protocol is very careful. Checkpoints are deployed byte-identical in counterbalanced blocks, the primary endpoint was locked and hashed before data collection, and exploratory results are clearly labelled. Since the claims are about rankings flipping, this is what makes me trust the results.

  2. The paper separates things that usually get lumped together: the effect of the collected states versus the warm start, with matched budgets, and ranking policies versus picking the best one. The live test makes a point I hadn't seen put this clearly: a proxy can rank policies well and still pick a worse deployed policy.

Critical weaknesses
  1. The headline result comes from one DAgger collection chain, and there's a plausible alternative explanation. The seeds only re-run fine-tuning on the same collected data (App. H), the margin between the two families is narrow next to BC seed variance, and the in-distribution reversal shows up on only one of the two validation clips. DAgger also roughly doubles the training data, and RQ3 shows that RGB drives most of the shift penalty: geometry-only BC on the demonstrations alone is about 4x better than the best DAgger checkpoint. So RQ1 may be more about giving an RGB policy more varied frames than about the collection regime, and once stronger candidates are added, the offline correlation turns positive (App. M).

  2. The setting is narrow and the link to deployment is mixed. There's one camera, one training actor, two held-out clips and one live setting. Under free zoom, the pre-registered primary endpoint favours the in-distribution choice, neither proxy ranks coverage well, and the 2x-floor sessions that favour the held-out choice are exploratory. The lesson in Sec. 8 is sensible but close to standard advice.

Questions for authors
  1. Is a second DAgger collection chain feasible? Without one there's no way to estimate collection-level uncertainty for RQ1.
  2. Could you add BC trained on the same number of extra frames collected independently of the learner? Sec. 5 names this control as missing.
  3. Does the inversion still hold if all ten candidates are geometry-only or use a frozen encoder? If not, I'd scope RQ1 to candidate sets with an appearance nuisance.
  4. Could you add statistics for the held-out clips to Table 5, and the held-out ranking with each clip dropped?
  5. Training longer on the demonstrations alone already lowers flat BC's held-out error a lot (App. L). Were the other BC candidates trained to convergence?
  6. Is participant video released, and does the consent cover that? The Ethics Statement also notes that no formal ethics-board review was done.

Other suggestions:

  • Sec. 7 says FOV-risk beats flat BC in all five rounds, but Table 11 shows a tie in round 2 (0.26 vs 0.26). Unrounded values would settle it.
  • Appendix G appears as an empty heading because Table 8 floats away from it.
  • Possibly relevant: Gulrajani and Lopez-Paz (ICLR 2021) on domain generalization, Miller et al. (ICML 2021) on accuracy on the line, Irpan et al. (NeurIPS 2019) on off-policy classification, and Wen et al. (NeurIPS 2020) on copycat agents in behavioral cloning.
1 comment
AuthorsOpenReview ↗

Round 2 follow-up: unseen people, resampling sensitivity and matched-row data

Follow-ups with the revised PDF.

One training person (critical weakness 2). We ran a post-review test on two previously unseen people, plus the original person in new clothing as a same-person control (general comment, new Appendix S). This adds external test people; training still used one person. Against BC trained for 60 epochs, DAgger-R2 had lower live J_0.25 for both new people: -0.17 (95% CI [-0.25, -0.08]) and -0.43 ([-0.50, -0.36]). Its offline error on their clips was also higher (0.123 vs. 0.117 and 0.235 vs. 0.044). The control showed the same reversal, so the result limits DAgger-over-BC across training budgets and cannot be attributed solely to testing unseen people. It is consistent with our round-1 answer to Q5: the paper's held-out error already ranked this BC checkpoint above DAgger-R2. At the 20-epoch budget of Table 2, DAgger-R2 beat L=1 and flat BC in 6/6 blocks for every person. We keep the limitation: one training person, one room per phase, one rig and one DAgger chain.

Q1, a second DAgger chain. A new chain is still not feasible. Appendix R.2 reports a conditional resampling sensitivity analysis, fixed before any training. It is not a second collection chain. Blocks of 300 frames from the collected data were resampled within each round, and the R1->R2->R3 fine-tuning chain was retrained 100 times. Within the original 20-epoch comparator set, DAgger-R2 had lower held-out error than all 21 BC-family checkpoints in 100/100 replicates. Its gap to the best of them has a 95% interval of [0.014, 0.092] (median 0.061). The full cross-family inversion held in 84/100 replicates, so it does not meet our pre-stated 90% robustness criterion. All 16 failures are R1 or R3 reaching the best BC checkpoint. Every resampled segment still comes from the one original collection trajectory and its policies, so the limitation stays in the paper.

Q2, data doubling. We also matched total rows (Appendix R.1, protocol fixed before training). Every arm adds 1,685 rows to the same base set (1,216 with a do command) and runs 8,568 updates. Held-out error is still lower with collected rows than with clip6 demonstration frames. The medians are 0.064 for round-3 rows and 0.045 for rounds 1-2, against 0.147, and each is better in 5/5 paired seeds. In-distribution error on the validation clip is higher (0.0010 and 0.0014 vs. 0.0005), the same inversion as in RQ1. Under these conditions, row count alone does not explain the held-out difference. These results do not identify which property of the collected data causes it (collecting policy, state distribution or corrections). The original DAgger-R2 held-out advantage over the compared 20-epoch BC checkpoints does not extend to the 60-epoch L=1 checkpoint, which already had lower held-out error. Post hoc, with the rule locked before computing (Appendix S), the collected-data arm trained from scratch at the 60-epoch budget (five seeds) had lower error on all three newly recorded clips (P0 control, P1 and P2) than the best 60-epoch BC recipe: medians 0.031, 0.022 and 0.057 vs. 0.102, 0.087 and 0.062. It was not deployed live and is not the chain recipe, so it does not show that DAgger beats 60-epoch BC in closed loop. In short, the new evidence does not supply multi-person training, an independent DAgger chain or a causal account of the collected data.

Reviewer_dvbKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether offline imitation error provides a reliable basis for comparing and selecting learned controllers for a real pan-tilt-zoom camera tracking a person. The policies imitate a deterministic expert whose actions depend on target-box geometry and camera pose.

The paper reports three main findings. First, in-distribution and shifted offline evaluation produce different orderings across BC and DAgger policies, although ordering within the BC family is unstable across optimization seeds. Second, BC trained from scratch on the aggregated DAgger dataset recovers most of the held-out improvement without the same increase in in-distribution error, suggesting that the collected data suffice without the particular warm-start training trajectory. Third, supplying RGB alongside the sufficient geometric inputs substantially increases shifted imitation error; several controls suggest that parameter count alone does not explain the effect.

A live test of the same ten checkpoints distinguishes top-one selection from overall ranking: the in-distribution-selected policy performs better than the held-out-selected policy on coverage and has the highest mean telemetry-based centred-tracking score, while held-out error ranks centred tracking across the full candidate set better. The physical experiments use one camera; an additional offline dataset check provides limited external evidence for the RGB finding.

Critical strengths

Useful controls separating collected data from training trajectory. Training BC from scratch on the DAgger-collected dataset, with both matched-epoch and approximately matched-update comparisons, provides evidence that the warm-start trajectory is not required to recover most of the held-out gain. This is more informative than a simple BC-versus-DAgger comparison.

Direct deployment testing with transparent qualification of results. Deploying the same checkpoints makes the distinction between selection and ranking concrete. The authors also distinguish pre-specified tests from exploratory analyses and report results that complicate their initial motivation. This transparency strengthens confidence in the narrow empirical conclusions.

Critical weaknesses

The broader significance of the findings remains insufficiently established. The physical evidence comes from one PTZ system, with one DAgger collection chain and a narrowly defined geometric expert. The external RGB check is useful but imprecise and does not validate the model-selection findings. Moreover, the stronger aggregate-from-scratch and geometry-only recipes are absent from the main live comparison, even though adding them changes the offline ranking picture. Consequently, it remains unclear how much the reported selection behavior reflects the particular candidate set and observation design, or what new actionable guidance follows for broader robot-learning settings beyond established principles.

The presentation obscures the contribution, while the explanatory account remains limited. The paper’s dense prose and numerous interleaved qualifications make it difficult to identify which experiments support which claims. Although Section 3.5 formally describes ranking reversals, its identity is descriptive rather than predictive. The RGB controls rule out some simple explanations but do not identify the learned mechanism. A controlled empirical characterization can be valuable without a new theory, but here the limited explanatory advance and narrow validation leave me unconvinced that the contribution is sufficiently substantial.

Questions for authors

What determined the exclusion of aggregate-from-scratch and geometry-only policies from the same-checkpoint live test? Given the candidate-set sensitivity reported in Appendix Q, how should readers delimit the deployment conclusions?

Which results support a claim about the general reliability of the held-out ranking proxy beyond the telemetry-based centred-tracking metric? Please distinguish those results from the coverage findings and the post-hoc measurement check.

3 comments
AuthorsOpenReview ↗

Round 2 follow-up: geometry-only and frozen-encoder policies deployed live

Follow-up after the post-review test on previously unseen people (general comment, new Appendix S).

Q1, geometry-only and frozen-encoder candidates live. The post-review test deployed a geometry-only and a frozen-encoder policy (seed 0, fixed before the session) alongside DAgger-R2, two L=1 BC policies (20 and 60 epochs) and flat BC. It is labelled post-review. On live J_0.25, geometry-only minus DAgger-R2 is +0.15 (95% CI [0.01, 0.31]; 4/6 blocks) for the original person in new clothing, -0.12 ([-0.25, 0.01]; 2/6) for new person P1 and +0.21 ([0.05, 0.37]; 5/6) for new person P2. Frozen-encoder minus DAgger-R2 is -0.06 ([-0.13, 0.01]), -0.17 ([-0.25, -0.10]) and +0.24 ([0.07, 0.41]). These deployments extend the live comparison to geometry-only and frozen-encoder candidates, but they do not restore the RQ1 inversion across candidate families. As stated in round 1, that inversion is limited to RGB candidates at the 20-epoch budget. Geometry-only had the lowest error on every new offline clip (0.037-0.042), yet the 60-epoch L=1 BC policy had the highest live J_0.25 for all three people. Top-one selection by that offline error would therefore have missed the best live policy here. Post hoc, the paper's held-out error also selects geometry-only among these six, while in-distribution error selects the 60-epoch policy (Appendix S, descriptive). The geometry-only and frozen-encoder policies are single seeds.

Q2, does held-out error rank live performance? With n = 6 checkpoints per person, the evidence is descriptive only. We took the Spearman correlation of each error, with its sign flipped, against live J_0.25, so a positive value means lower error goes with higher J_0.25. The values are +0.66, +0.54 and +0.83 for the paper's held-out error and +0.26, +0.26 and +0.54 for in-distribution error. Offline, over the ten Table 2 checkpoints, our pre-stated rule supports the held-out ranking on one new person's clip and not the other's (19/21 vs. 4/21 BC/DAgger pairs keep the held-out order). We therefore make no general reliability claim.

Thanks for the detailed response and the additional experiments. I appreciate the effort the authors have put into addressing my concerns.

The new experiments with previously unseen participants, the additional policy comparisons, and the matched-data controls are helpful. I also appreciate that the authors have clarified the scope of their claims and acknowledged the limitations of the original results. These additions make the empirical study more convincing.

However, my main concern is still about the broader significance of the findings. The results seem quite sensitive to the particular candidate policies, training budgets, and evaluation conditions. For example, the BC versus DAgger comparison changes when BC is trained for longer, and the offline metric that provides better overall ranking does not necessarily select the best-performing policy. These observations are interesting, but I am not sure what general conclusions we can draw from them.

I also think the paper still lacks a clear explanation of why these observations occur. The additional controls help rule out some possible explanations, but they do not really establish the underlying mechanisms. Without a more general analysis or evaluation across different robotic systems, it is difficult to tell whether the findings are specific to this PTZ setup or reflect something more fundamental about imitation learning under distribution shift.

Finally, while I appreciate the authors' efforts to clarify the presentation, I still feel that the paper is unnecessarily complicated for the main messages it conveys. The distinction between offline ranking and selecting the best policy is useful, but I think the paper could do a much better job of highlighting a few clear takeaways rather than presenting many narrowly defined experimental observations.

Overall, I think the rebuttal has improved the paper, especially in terms of experimental rigor. However, I am still not fully convinced that the findings provide sufficiently significant or generalizable insights beyond this particular system. I will keep my current rating.

AuthorsOpenReview ↗

Three takeaways

Thank you for the careful follow-up. We agree that the study does not establish mechanisms and covers one PTZ system; the paper states both. On what general conclusions can be drawn, we see three, each scoped to what was tested:

  1. Ranking quality and selection quality are different properties. Among the six deployed checkpoints, held-out error ordered more pairs correctly than in-distribution error (11, 11 and 13 of 15 vs. 9, 9 and 11), yet its top-one pick was not the best live policy (regret 0.03, 0.31 and 0.19 in median J_0.25). Reporting rank correlation alone can hide such selection failures.

  2. The DAgger-over-BC conclusion reverses when BC is trained longer. DAgger-R2 beat both deployed 20-epoch BC policies in 6/6 blocks for every person, and lost to 60-epoch BC for all three.

  3. Offline error is best used as a shortlist, followed by a short closed-loop check at deployment settings. This is a recommendation from these results, not a procedure we validated.

Whether these hold beyond this PTZ setup is untested, and we do not claim it. If accepted, we will restructure the abstract and introduction around these three takeaways and move the narrower observations to the appendix.

Reviewer_DVBnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies when per-step imitation error on logged states is a valid proxy for selecting among imitation-learned controllers, using a single physical pan-tilt-zoom camera for indoor single-target tracking. The expert is a deterministic center-lock controller that depends only on the target box and camera pose, so it can be queried at any visited state, which enables both behavior cloning and DAgger. The main contributions are three findings. I have organized them as research questions-answers. For RQ1, in-distribution imitation error and held-out-shift imitation error induce opposite orderings of BC and DAgger policies. For RQ2, plain BC trained from scratch on the union of the demonstration clips and the DAgger-collected states recovers most of DAgger's held-out fidelity without its in-distribution penalty, under both matched-epoch and matched-update budgets, so the authors conclude the collected support is sufficient and the warm-start trajectory is not required. For RQ3, removing the RGB branch reduces held-out error 10-17x, and parameter-matched, frozen-encoder, constant-image, and shuffled-image controls indicate the cost arises from state-aligned visual content rather than parameter count

Critical strengths
  1. The methodological discipline is exemplary for a small-n physical study and is, in my view, the paper's most important contribution. The threats-to-validity table (Table 8) addresses the objections I would have raised, including that the proxy favors BC by construction. Because the field has a replication problem precisely in closed-loop robot evaluation, a paper that shows how to report such a study credibly has value beyond its specific findings.

  2. RQ2 and RQ3 are both practically useful and properly controlled. The matched-update control rules out the explanation that the support effect is an artifact of extra optimizer steps, and the observation-control grid in Appendix is a great set of ablations that are needed to localize the RGB cost.

Critical weaknesses
  1. The live result complicates the paper's thesis more than the framing acknowledges. The abstract and introduction argue that logged-state imitation error is not a stable ordering proxy, yet in the pre-registered live test the in-distribution choice (L = 1) achieved the highest mean J_0.25 of all ten policies and more coverage than the held-out choice in 5/5 blocks, falling in the authors' own locked category of no mis-selection detected. The authors reconcile this via the selection-versus-ranking distinction, which is a fair and interesting point, but a practitioner reading the abstract would conclude in-distribution error is unreliable when in this experiment it selected the best deployment policy.

  2. The scope of the evidence is narrow in ways that limit what can be concluded. All results come from one camera, one DAgger collection chain one actor and one room in the live test, and a task with a 3-DOF output and a deterministic geometric expert. RQ3 in particular is close to guaranteed by construction: since the expert is a function of box and pose alone, a geometry-only policy is appearance-invariant by design. Whether any of the three findings transfer to settings where the expert actually depends on visual input, which is the common case in imitation learning, is I feel untested.

Questions for authors
  1. Given that the in-distribution choice was best in the live test, what do the authors recommend a practitioner do when deploying a single checkpoint? (The paper currently argues both sides).
  2. OPED correlates +0.79 with the held-out ranking. How does it correlate with the live J_0.25 and coverage rankings?
  3. Could the authors report the live comparison restricted to ticks at matched zoom, or replicate the 2× floor across several blocks, to separate tracking from zoom policy?
2 comments
AuthorsOpenReview ↗

Round 2 follow-up: checkpoint selection after the post-review test

Follow-up after the post-review test on previously unseen people (general comment, new Appendix S).

Q1, selecting one checkpoint. The new six-checkpoint test shows that single-checkpoint selection outcomes differed with the eligible training budget and the deployment setting (the test does not isolate either factor). At the 20-epoch Table 2 budget, DAgger-R2 outperformed the in-distribution choice (L=1, 20 epochs) on live J_0.25 by +0.27, +0.22 and +0.19 for the three people. It won 6/6 blocks per person (nominal sign-flip p = 0.031 within each person), although the coverage intervals include zero. Once a 60-epoch BC checkpoint was eligible, that checkpoint beat DAgger-R2 for all three people, and both offline errors had already preferred it. All runs used a 2x zoom floor, not the free-zoom protocol, so they do not revise the free-zoom result that in-distribution error selected L=1. Our recommendation therefore follows from this sensitivity: treat offline error as a shortlist and confirm the choice with a short closed-loop check at the deployment settings. Descriptively and post hoc (Appendix S), among these six checkpoints in-distribution error selects L60, the best live policy for every person, whereas the paper's held-out error selects geometry-only (regret 0.03, 0.31 and 0.19 in median J_0.25), even though held-out error orders more pairs correctly (11, 11 and 13 of 15 vs. 9, 9 and 11). Higher rank agreement therefore did not give better top-one selection here. This is not evidence that in-distribution error is generally better (n = 6).

Scope. The post-review test did not confirm a DAgger-over-BC ordering against 60-epoch BC, for either new person or the control. The scope stays at one training person, and the abstract and limitations now report this result.

Thank you for the response, which clarifies and further strengthen the paper's methodological rigor. However, the practical conclusions remain sensitive to training budget and deployment conditions, and the narrow experimental setting still limits generalizability, so I would like to keep my score unchanged.

All the best!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Round 2: post-review test on previously unseen people

After round 1 we ran a live and offline test on two people absent from every training and evaluation clip (P1, P2). P0, the original person in clothing not used before, is a same-person control: same room and camera as the paper's live sessions, a different day, and furniture since moved. The protocol, checkpoint hashes, block order, analysis script and scenario rule were locked before any new-person data existed (new Appendix S). This is post-review evidence, not pre-planned relative to the submission. Each person ran six blocks over six frozen checkpoints in a Williams design: 75 s rollouts, with the 2x zoom floor and route of Appendix P, not the free-zoom protocol. Each person also walked one 150 s scripted clip for offline scoring. The unit is the block within a person, and people are not pooled. All 108 slots are valid.

Primary comparison 1: DAgger-R2 vs. BC L=1 trained for 60 epochs (L60), fixed before the session as the BC comparator with the lowest held-out error. The pre-stated rule classifies it as reversed. Live J_0.25, DAgger-R2 minus L60, was -0.17 for P1 (95% CI [-0.25, -0.08], 1/6 blocks won by DAgger-R2). It was -0.43 for P2 ([-0.50, -0.36], 0/6) and -0.28 for P0 ([-0.39, -0.17], 0/6). Offline error on each person's clip was 0.123 vs. 0.117 (P1), 0.235 vs. 0.044 (P2) and 0.206 vs. 0.097 (P0). DAgger-R2 therefore does not dominate BC across training budgets. The reversal also occurs for P0, so an unseen person is not necessary for it.

Our pre-written S3 template described this outcome as a failure of the paper's held-out ranking. For these two checkpoints that sentence was wrong. The paper's held-out error (0.056 vs. 0.073) and in-distribution error (0.0027 vs. 0.0208) both preferred L60, matching the live order. The pair contradicts an unrestricted DAgger-over-BC claim but does not distinguish the two proxies, both of which predict this pair's live ordering. Appendix S records this discrepancy. The RQ1 inversion stays restricted to RGB candidates at the 20-epoch budget, and the abstract and limitations now report the reversal.

At the 20-epoch Table 2 budget (secondary comparisons), DAgger-R2 beat the in-distribution choice L=1 20ep on J_0.25 by +0.27, +0.22 and +0.19 (P0, P1, P2). It beat flat BC by +0.26, +0.29 and +0.23. Each of these is 6/6 blocks within each person (nominal sign-flip p = 0.031, the minimum with six blocks; within-person, not a population test). Coverage does not confirm the L=1 20ep difference, since all three CIs include 0.

Primary comparison 2 (geometry-only minus DAgger-R2, J_0.25) is mixed. It is +0.15 for P0 ([0.01, 0.31], 4/6), -0.12 for P1 ([-0.25, 0.01], 2/6) and +0.21 for P2 ([0.05, 0.37], 5/6).

Rank agreement (pre-specified item 3, n = 6 live checkpoints per person, descriptive). The locked script did not fill this field because its reference file was absent. We applied the same function after the session, in a separate file, to each error with its sign flipped, so a positive value means that lower error goes with higher live J_0.25. The values are +0.66, +0.54 and +0.83 for the paper's held-out error and +0.26, +0.26 and +0.54 for in-distribution error. These rankings are not top-one selection, and n = 6 is small. Offline over the ten Table 2 checkpoints, the pre-stated rule supports the paper's held-out claim on P1's clip (held-out rho +0.64 vs. in-distribution -0.39; 19/21 BC/DAgger pairs keep the held-out order). It does not support it on P2's clip (-0.19 vs. +0.61; 4/21). The general reliability of held-out error as a selector therefore remains unresolved. Appendix S adds two post-hoc analyses with rules locked before computing: top-one selection among the six checkpoints, and collected data at the 60-epoch budget scored offline on the new clips.

Protocol fidelity. Eight of 116 attempts were replaced, all for faults the operator classified as exogenous under the pre-set limit of one per slot and four per person. Four were a second person entering the room, two were the actor leaving the route, and two were a missed audio cue. After the second missed cue, the browser cue page was replaced by a laptop-side player with the same cue times and words, and its log records no missed cue. One replacement was run later in the day. No computed scores or policy comparisons were inspected before all sessions ended. Only the operator classified failures; we have no independent second review of the videos.

Summary. The restricted 20-epoch DAgger advantage held in both new people at this zoom setting, but the broader DAgger-over-BC conclusion did not hold against 60-epoch BC. Neither result shows that held-out error is a universally reliable selector. The test covers two people and one control in one room, with one rig and one DAgger chain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.