- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TRIAL, which evaluates active discovery of hidden temporal rules in procedurally generated worlds. Agents construct experiments through navigation and material manipulation, observe partial feedback, and make an irreversible rule commitment. The experiments compare frontier agent systems with and without solver code and file memory. The authors found that no system identifies a rule on the hardest condition. The paper makes several analyses. For example, Exact-answer scoring hides the guesses that happen to be right.
Critical strengths
- The benchmark provides precise, turn-level diagnostics of information acquisition. It helps for fine-grained experimental analyses. Also, the difficulty is controlled through hypothesis-space size and experiment cost.
- The paper introduces a concrete planning challenge. Requiring agents to construct experiments through environmental actions also introduces a meaningful trade-off between information gain and execution cost. The trajectory-level analysis, including hypothesis elimination, remaining uncertainty, etc., offers more diagnostic information than a binary success rate.
Critical weaknesses
-
Insufficient validation of task feasibility and the experimental planning baseline. The paper states that
no system identifies a rule on the hardest condition.However, a major concern is whether the harder benchmark conditions are reliably identifiable within the allocated interaction budget. The generator guarantees that an accepting state is reachable, but this is not equivalent to guaranteeing that the hidden rule can be uniquely identified within 240 decisions. Without such a reference, it is difficult to determine how much of the observed failure reflects inadequate experimental planning capabilities rather than insufficient budget or limitations of the task interface. -
Limited depth of the empirical insights. The main empirical findings are largely consistent with prior expectations. For example, correct answers may remain underdetermined by the collected evidence, agents may exhaust their budgets before resolving uncertainty and then submit wrong answers, and self-reported uncertainty may deviate from the evidence-based reference, etc. These observations are reasonable but they seem not to provide substantially new insights into the mechanisms underlying failures in agentic scientific experimentation.
Questions for authors
- Could the authors bring more specific analyses on whether the primary bottleneck lies in experiment selection, long-horizon planning, cost estimation, belief tracking, etc.? The mechanism-level analyses would make the observations of the work more insightful. I would find the empirical contribution substantially stronger if the benchmark were used to uncover more specific, non-trivial failure mechanisms, rather than primarily confirming anticipated consequences on budget-constrained hypothesis identification.
- The greedy baseline in Section 4.3 already uses the exact posterior and information gain per action. However, it does not establish whether the hardest main-benchmark targets can be uniquely identified within the allocated budget. Could the authors provide budget-feasible identification trajectories or results from a stronger planning reference for these targets?
We thank the reviewer for the thorough review.
Addressing the questions:
primary bottleneck
We tried to focus on leveraging the structural qualities of the task design for evaluation rather than using black box evaluation to score the large volume of data from the benchmark.
budget constraints
This is a good point. Our results show that the agents are able to solve even the hardest environments. The additional results for reviewer 2 show that, as expected, increasing the budget increases the success rate of the agents.
As noted by the reviewer, this dynamic is expected and not unique. We want to highlight our core contribution. Unlike other benchmarks and exploration evaluation environments, where a semi-formal rule is encoded into an environment through a rubric, we begin with a strict formal representation of a rule in DFA form. By adding arbitrary epsilon moves to the DFA, we can easily construct easy (a few epsilon moves) and hard (more epsilon moves) environments that have different syntactic and semantic difficulty whilst remaining structurally identical.
One episode starts in the easy environments. These have very low noise, with some degree of exploration; the agent should be able to brute-force these cheaper environments. As the agent solves the environments, the benchmark progressively loads in harder games that still obey the same rule. Once the agent believes that it has understood the underlying rule, it can commit its hypothesis to the scorer, which can deterministically verify the hypothesis.
The ramp-up nature of the tasks is designed to penalize brute-force strategies. We find that even frontier reasoning agents tend to fall into these patterns.