- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work presents DexHoldem. It decomposes Texas Hold’em tabletop interaction into 14 manipulation primitives on a ShadowHand–UR platform, providing a series of teleoperated demonstrations, a physical policy benchmark, a state-perception benchmark, and closed-loop system evaluation. The authors also compare nine policies. The main contribution is a benchmark and empirical diagnosis, which is meaningful for the research community.
Critical strengths
S1. This work presents a useful real-hardware benchmark, which is overall comprehensive.
S2. Two-axis scoring exposes side effects hidden by completion alone. Separating task completion from scene preservation, with explicit non-target displacement and overlap criteria, helps assess precision in sequential tabletop manipulation.
Critical weaknesses
W1. Perception scoring omits key runtime information, this limitation weakens the benchmark’s measurement validity.
W2. Ranking stability is insufficiently established. Most primitives receive only five physical trials. The three perception runs reuse the same 36 problems, and the top two configurations differ by just one of 108 attempts. Tables 1–2 provide neither confidence intervals nor variation across policy-training seeds in physical evaluation. The reported differences are descriptive, leaving the stability of benchmark rankings unresolved.
W3. Closed-loop evaluation lacks controls for component attribution. The 33 hands evaluate only GPT 5.5 with π0, without controlled component substitutions or a recovery-disabled condition. The 12/34 successful retries demonstrate that some stalls can be resolved, but do not quantify recovery’s net benefit or separate perception, execution, and routing contributions to hand-level failure. The authors also acknowledge the need for these controls.
W4. A failed pickup automatically becomes a failed paired put-down even when the placement policy is not executed, while Table 11 combines put-down/show. These scores characterize a sequential chain but cannot isolate placement competence. Report both put-down success conditional on successful pickup and independent results from a standardized held-card initial state.
Questions for authors
Q1. Could the authors report held-card rank/suit and scene_stable accuracy, and compare each model with current-image-only input versus annotated-history input? The predecessor-copy baseline does not replace a model-specific history ablation.
Q2. Could the authors compare recovery enabled versus disabled, and an alternative perceiver or ground-truth state input, under matched initial states and common termination rules? Please report completion without human-help requests and failure counts by component. These comparisons would materially affect the assessment of the closed-loop benchmark’s diagnostic value.
Q3. Please report actual execution counts and conditional success rates separately for each put-down and show primitive, and clarify how show trials are scored after a failed pickup. Would an independent evaluation from a standardized held-card state change the comparison of placement competence?
2 comments
Weakness3: Closed-loop evaluation lacks controls for component attribution. The 33 hands evaluate only GPT 5.5 with π0, without controlled component substitutions or a recovery-disabled condition. The 12/34 successful retries demonstrate that some stalls can be resolved, but do not quantify recovery’s net benefit or separate perception, execution, and routing contributions to hand-level failure. The authors also acknowledge the need for these controls.
Question2: Could the authors compare recovery enabled versus disabled, and an alternative perceiver or ground-truth state input, under matched initial states and common termination rules? Please report completion without human-help requests and failure counts by component. These comparisons would materially affect the assessment of the closed-loop benchmark’s diagnostic value.
We agree that the 33 hands evaluated with GPT 5.5 and π0 do not isolate the contributions of perception, execution and routing. Expanding controlled system-level evaluation is currently costly: a complete evaluation session takes over an hour, and perception errors and policy failures frequently require human intervention. These constraints limit the scale of evaluation, but do not resolve the attribution question.
Three of the 33 hands completed without human help, including two that used retries, and 12 of 34 retries completed the stalled primitive. Under a recovery-disabled protocol that terminates at the first stall, truncating the recorded trajectories leaves one unassisted completion out of 33.
Weakness4: A failed pickup automatically becomes a failed paired put-down even when the placement policy is not executed, while Table 11 combines put-down/show. These scores characterize a sequential chain but cannot isolate placement competence. Report both put-down success conditional on successful pickup and independent results from a standardized held-card initial state.
Question3: Please report actual execution counts and conditional success rates separately for each put-down and show primitive, and clarify how show trials are scored after a failed pickup. Would an independent evaluation from a standardized held-card state change the comparison of placement competence?
We do not evaluate put-down independently because it requires the policy to first pick up the card. Providing a pre-held card would bypass this requirement and would not reflect the policy’s performance in the actual task. Moreover, the pickup position is not fixed, so the starting state for put-down depends on the policy’s own grasp. We therefore evaluate put-down after the policy’s own pickup and count a failed pickup as a failure of the paired put-down. The same rule applies to show.
Question1: Could the authors report held-card rank/suit and scene_stable accuracy, and compare each model with current-image-only input versus annotated-history input? The predecessor-copy baseline does not replace a model-specific history ablation.
We now report held-card rank-and-suit exact match, scene-stability balanced accuracy (BA), and model-specific history ablations.
All values below are percentages. “With / without” refers to annotated history. Both conditions retain current and past images. The held-card test uses only the current image.
| Configuration | Held-card exact match | Overall: with / without annotations | Stability BA: with / without annotations |
|---|---|---|---|
| GPT 6 Astra | 100.0 | 43.5 / 38.9 | 76.9 / 75.3 |
| GPT 6 Sol | 90.9 | 39.8 / 36.1 | 82.3 / 77.2 |
| GPT 5.6 Sol | 81.8 | 49.1 / 25.9 | 89.2 / 73.8 |
| GPT 5.6 Luna | 69.7 | 28.7 / 14.8 | 85.5 / 71.6 |
| Opus 5.5 | 97.0 | 49.1 / 36.1 | 90.3 / 81.9 |
| Sonnet 5 | 100.0 | 36.1 / 21.3 | 88.7 / 79.6 |
| Opus 4.7 | 100.0 | 34.3 / 20.4 | 77.7 / 58.6 |
| Sonnet 4.6 | 90.9 | 25.0 / 13.9 | 91.4 / 50.1 |
| Haiku 4.5 | 0.0* | 13.9 / 4.6 | 74.1 / 46.8 |
| Gemini 3 Flash | 72.7 | 20.4 / 13.0 | 82.7 / 71.9 |
| Gemini 3.1 Flash Lite | 100.0 | 10.2 / 14.8 | 47.8 / 48.9 |
The held-card test uses 11 images; the state evaluation uses 36 scenes. Each is repeated three times.
We agree that the original perception aggregate did not directly score all information required during execution. Held-card recognition and scene-stability checks are already used in our system, and we now report their accuracy separately.
Held-card recognition is evaluated on 11 images over three runs, using only the current image. Scene stability is evaluated on the 36 original scenes with annotated history. We additionally report the without-annotated-history results in our responses to the other reviewers. These evaluations extend coverage to the omitted runtime fields, and we will clarify which capabilities the original aggregate measures and which require separate diagnostics.
The evaluated GPT, Claude, and Gemini configurations use provider-hosted models, so their wall-clock latency includes model inference, provider-side serving, tool execution, and network overhead. This makes it difficult to compare inference efficiency under a shared hardware setup. Our perception scores therefore measure state-recovery accuracy. Deployment latency remains practically relevant, and we acknowledge that the current benchmark does not provide a controlled efficiency comparison.
We added 320 physical trials, repeating the full evaluation for π0.5, π0, RDT, and DP (DINO) with the same checkpoints and protocol. Each now has 160 trials: 20 per pickup primitive and 10 per other primitive.
Physical results, with 95% Wilson confidence intervals. SPSR and TCR are reported in percent.
For these checkpoints, both π policies outperform RDT and DP (DINO) in SPSR (all four pairwise Fisher tests: p ≤ 0.006), but the difference between π0.5 and π0 is not significant (p = 0.65). These repeated evaluations measure physical execution variability; variation across policy-training seeds remains unmeasured.
For perception, three runs still cover only 36 scenes. We therefore report paired comparisons with confidence intervals obtained by resampling scenes, rather than treating all 108 attempts as independent.