- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper proposes EviStateBench, a benchmark for evaluating whether embodied agents can maintain temporally consistent, evidence-aware task-state beliefs from noisy, delayed, incomplete, and contradictory observation streams. The main contributions are a public-input/hidden-oracle evaluation protocol, a generation pipeline that produces clean and perturbed streams from BEHAVIOR-style simulator executions, a workload of eight query families and a reference system, EviStateDB, that maintains bitemporal state with supporting and contradicting evidence.
Critical strengths
The problem the benchmark targets is real and under-served. The paper correctly observes that existing embodied benchmarks does not isolates whether an agent can answer what state held at a valid time, what was knowable before late evidence arrived, or which observations support a belief.
Critical weaknesses
I have one critical concern which is that the baselines do not establish that the benchmark is hard for any method a practitioner would actually deploy. All three are hand-written systems, and two of them implement essentially the same bitemporal data structure from which the oracle is derived. Their 98% exact accuracy on AS_OF_STATE and WHY_STATE indicates those query families are solved by implementing the obvious representation. Additionally, no learned memory system and no LLM-based agent is evaluated, even though these are some obvious intended users of the benchmark. Furthermore, the observation streams are symbolic predicates with injected noise rather than outputs of any perception stack, so I feel that the "embodied" framing is a motivation and not tested property. Together these points leave me unsure what a score on EviStateBench currently tells us about an embodied agent.
Questions for authors
- For STATE_DIFF and FIND_UNCERTAIN_STATES, is the hidden answer unique under reasonable alternative temporal semantics? What fraction of queries would change answer under, for example, open versus closed interval conventions?
- Is the retrieval baseline intended as a strawman? What does a retrieval system with time-aware ranking score?