- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces HiddenCoop, a benchmark in which cooperation requirements are initially hidden and become known through valid interactions. It proposes Discovery-Probability Graph Planning, which separates confirmed facts from beliefs about undiscovered resource counts and locations. Exploration and transport candidates are evaluated on shared sampled worlds using deterministic joint search, and plans are revised as observations and messages arrive. Evaluation uses 50 matched ordinary/cooperative task pairs in C-WAH and 24 in TDW-MAT, with three runs per C-WAH task and one per TDW-MAT task. Cooperative-task success reaches 99.3% and 58.3% at the respective 200-second and 20-second embodied execution budgets. Additional analyses cover budget cutoffs, common-success efficiency, target completion, and model usage.
Critical strengths
-
Hidden cooperation is defined through an explicit information and execution protocol. The distinction between confirmed facts, unknown requirements, and sampled hypotheses makes the task reproducible in principle and avoids treating hypothetical discoveries as observed state. This is a useful contribution to evaluating cooperation under partial observability.
-
The full-system evaluation is comparatively careful. Matched ordinary/cooperative tasks, multiple budget cutoffs, common-success comparisons, and separate goal-level analyses help distinguish eventual completion from execution efficiency. The strong C-WAH results and low recorded model usage support the value of the complete framework within the stated interface. EET explicitly excludes model and planning wall-clock time, so the efficiency evidence should be interpreted within that definition.
Critical weaknesses
-
The central probability mechanism is not isolated by a matched ablation. The appendix explicitly treats the single-point comparison as a contrast between system versions, and the illustrative traces differ in model-call counts and other behavior. The 29.1% EET reduction therefore cannot be attributed to the probability representation alone. The full-system comparison remains useful, but the specific methodological advance over alternative discovery and planning strategies is less firmly established.
-
The strength of the TDW-MAT advantage is uncertain. Cooperative success of 58.3% versus PCE's 45.8% corresponds to 14/24 versus 11/24 tasks in a single run. Overall success is tied at 64.6%, and ordinary-task success is lower. The reported source-pair bootstrap principally estimates ordinary-to-cooperative changes, not the paired difference between methods. This limits the claim of robust superiority across both environments, although it does not negate the stronger C-WAH evidence.
Questions for authors
-
Clarification of the probability comparison. What are all behaviorally relevant differences between the probability and single-point versions, including candidate generation, replanning triggers, semantic requests, and execution? Is there an existing comparison in which these components and budgets are held fixed and only the belief representation and its evaluation change?
-
Clarification of the TDW-MAT results. Can you provide a paired success/failure cross-tabulation for DPGP versus PCE and an interval for their cooperative-task success difference? Are additional runs available, or should this result be interpreted strictly as a descriptive comparison on the single fixed run set?
2 comments
Response to Reviewer 8Tqi — Part 2/2
2. Additional TDW-MAT experiments (W2, Q2)
We added two full runs for each method, DPGP and PCE, to the original run. We evaluated both methods on the same 48 tasks in all three runs, yielding 144 episodes per method. We used different random seeds across runs. Within each run, both methods used the same environment settings and seed for each task. The results below use the 20 s budget. Overall, DPGP achieved higher task success (68.8% vs. 63.2%), lower mean EET (12.09 vs. 14.34 s), and higher TGC-AUC (0.6325 vs. 0.5474) than PCE.
&\text{Task SR (\\%)}\uparrow&\text{EET (s)}\downarrow&\text{TGC-AUC}\uparrow&\text{Tokens (k)}\downarrow\\\\ \hline \begin{array}{l}\text{Method}\\\\\text{PCE}\\\\\text{DPGP}\end{array}& \begin{array}{ccc}\text{All}&\text{Ord.}&\text{Coop.}\\\\63.2&\mathbf{84.7}&41.7\\\\\mathbf{68.8}&76.4&\mathbf{61.1}\end{array}& \begin{array}{ccc}\text{All}&\text{Ord.}&\text{Coop.}\\\\14.34&11.40&17.27\\\\\mathbf{12.09}&\mathbf{10.03}&\mathbf{14.16}\end{array}& \begin{array}{ccc}\text{All}&\text{Ord.}&\text{Coop.}\\\\0.5474&0.6431&0.4518\\\\\mathbf{0.6325}&\mathbf{0.6929}&\mathbf{0.5721}\end{array}& \begin{array}{c}\text{All}\\\\1128.53\\\\\mathbf{1.41}\end{array} \end{array}$$ On cooperative tasks, DPGP succeeded in 44/72 episodes (61.1%) and PCE in 30/72 (41.7%), a difference of 19.44 percentage points. Using an episode-level normal approximation for the paired success-rate difference across the 72 episode pairs, we obtained a 95% confidence interval of [4.22, 34.67] percentage points. The paired cooperative-task outcomes are shown by run below. |**Run**|**Both succeeded**|**Only DPGP succeeded**|**Only PCE succeeded**|**Both failed**|**Pairs**| |:---|:---:|:---:|:---:|:---:|:---:| |run_0|7|7|4|6|24| |run_1|7|8|3|6|24| |run_2|6|9|3|6|24| |Total|20|24|10|18|72|
Response to Reviewer 8Tqi — Part 1/2
Dear Reviewer 8Tqi, we thank you for your time, effort, and insightful comments. Please find our responses below.
1. Differences from the single-point version and additional component ablations (W1, Q1)
Thanks for pointing this out. We acknowledge that our previous ablation did not clearly isolate the influence of the probability mechanism. The single-point version requests definite target locations from the LLM, whereas DPGP uses LLM-derived semantic support to construct count and location distributions. The original 29.1% EET reduction compares the complete probability-based and single-point planners; the new ablations examine the three components separately.
We examine three components of probability-based planning: probability correction (P), multi-scene sampling (S), and joint candidate evaluation (E). The table below compares DPGP, the original single-point version, and the three component replacements.
We evaluated the three replacements on the same C-WAH tasks with the same seeds and environment execution budgets. All three retain DPGP's communication, memory update, replanning triggers, and execution procedures, while changing the specified component.
The following results report mean EET (s) for DPGP and the three replacements at six execution budgets.