acceptodds
Under review as a conference paper at ICLR 2027

BodyReady: Grounding Human-Centere Planning in Visual Preconditions and Action Effects

Abstract

Human-centered planning and assistance require more than predicting a plausible next action: they also require knowing whether the observed person can actually begin that action from the current bodily state. Existing benchmarks largely evaluate task progression, symbolic action preconditions, or agent and object feasibility, but do not isolate whether a specific person's visible bodily state permits an action or how preceding actions change what becomes possible next. Instead of treating action plausibility as sufficient, we explicitly ground action applicability in the person's hand use, posture, and target reachability, and evaluate how action-induced changes in bodily state alter subsequent readiness. We introduce , a benchmark for visual action applicability of a specific human, comprising 1,190 questions grounded in 280 exocentric images. Six complementary tasks evaluate whether an action can begin, what prevents it, which preparation can enable it, and where a short action sequence first becomes invalid. Reference answers are derived from human-reviewed visual states, explicit action prerequisites, and bounded completion effects. Across 18 proprietary and open VLMs, the best model achieves 74.96% strict overall accuracy, but only 63.85% on selecting currently applicable actions. Even when it correctly identifies an action as unready, it fails to diagnose the correct set of bodily blocker categories in 28.88% of those cases. Models also choose preparations that cannot themselves begin and fail to identify the first invalid step when the same actions are reordered. These results reveal a gap between predicting plausible actions and reasoning about whether a particular person can execute them from the current bodily state, establishing visually grounded human action applicability as an important capability for planning and assistance.

Primary area:
datasets and benchmarks
Keywords:
LLM, Human, Planning, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Aht7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In the paper, the authors first formulate action applicability (vision-based and human-specific) and introduce the BodyReady benchmark for the visual action applicability of a specific human, which evaluates whether a body state enables an action. The benchmark is built on 280 exocentric images and contains 1,190 questions. There are 6 tasks evaluating different aspects of action readiness, and they evaluate 18 VLMs (both closed- and open-source). They find a gap between the execution possibility of an action and its plausibility.

Critical strengths
  • Clear formulations and tasks.
  • Comprehensive evaluation (multiple different VLMs) and rigorous diagnosis of the unmet body conditions.
Critical weaknesses
  • Generalization to other embodiments: Robotic hands, humanoids, and similar entities have different body structures and different sets of possible actions due to differences in angles, joints, heights, and so on. Formulations are usually general first and then narrowed down to a specific setup. Can the formulation of action applicability and readiness be extended to these other embodiments?
  • Motivation and Application: If the goal is for a robot to execute a sequence of applicable actions toward a goal, why would the vision be from an exocentric point of view, and why is it human-centric? These cases seem applicable only when there is a need to train a robot or agent from human actions, or for human-assistance applications. Therefore, since the benchmark acts as a test set, can this evaluation framework be utilized for test-time self-critique to enable replanning loop before action execution?
  • Narrow data collection setups: The BodyReady data collection process introduces several narrow setups:
    • Static action matching: Because the analysis is performed on individual, static frames, it limits the ability to expand the task to include the temporal axis (e.g., pace, duration).
    • Object detection: Objects are detected using bounding boxes rather than semantic masks. This limits tasks that might relate to the feasibility of holding challenging shapes (e.g., narrow, sharp, or overly large objects) or interacting with object materials and additional attributes (e.g., cold, hot, or slippery surfaces).
    • Limited camera angles.
Questions for authors
  • How complex are the visual scenes in terms of object density, distractors, and so on? Are there statistics regarding the complexity of the data (both the scenes and the task instructions)?
  • How is reachability assessed? The distance should be calibrated by the relative locations of the person and the target object with respect to the camera location.
7 comments
AuthorsOpenReview ↗

Response to Reviewer Aht7 (1/4)

Thank you for the positive assessment and the thoughtful questions. We appreciate your recognition of the clear formulation and tasks, and of the comprehensive evaluation.

W1: Extension to other embodiments

Thank you for this question. Yes, the formulation extends to other embodiments: its logic does not depend on the human body, and only its content must be redefined for each body.

Part of the formulationFor another embodiment
Readiness: all requirements of an action are met (Eq. 2)Unchanged
Blockers: the channels whose requirements failUnchanged
Completion effects: how an action updates the stateUnchanged
What the hand, posture, and reach channels recordRedefined
Each action's requirements R(a)Redefined

For example, for a two-armed mobile robot, the hand channel would record what each gripper holds, the posture channel the configuration of its base and torso, and the reach channel what its arms can reach from the current base position. "Pick up the box with both hands" would then require two free grippers and the box within reach. The angles, joints, and heights you mention enter through the posture and reach channels and through R(a); the computation of readiness, blockers, and effects stays the same.

We start with humans because assistance requires judging the person, and because the human body has constraints, such as posture transitions and the coordination of two hands, that need dedicated definitions. Our data cover only humans, so we present the extension at the level of the formulation. The revision adds a short paragraph, "Beyond human bodies", with this table.

AuthorsOpenReview ↗

Response to Reviewer Aht7 (2/4)

W2: Motivation and application

Thank you for raising this. Assistance and human–robot collaboration, the second case you list, are the target setting of BodyReady. For example, a robot judges whether this person can begin an action now, and uses that judgment to decide whether and how to help. By human-centered planning in the title, we mean planning by a robot or assistive system of how to work with a person, including which steps the person performs, which the robot performs, and when to intervene. The exocentric view follows from this setting. An assistant observes the person from the outside, whereas an egocentric view shows little of the wearer's own body.

Test-time self-critique. Yes. Before a plan is executed, BodyReady's tasks can serve as the checks of a self-critique and replanning loop. The loop checks whether each step can begin, finds where the plan first fails, and diagnoses what blocks that step. It then revises the plan by inserting a preparation or a help action that lets the step begin, and repeats the checks on the revised plan. Each check can be run and scored on its own, so the benchmark measures how reliable each check would be.

Step of self-critique before executionBodyReady taskClaude Opus 5.5
Can this step begin?AR / AS82.4% / 63.8%
Where does the plan first fail?SV79.9%
What blocks it?BD65.7%
Which preparation would let it begin?PS83.8%

Even the strongest model stays below 85% at every step, so such self-critique cannot yet be trusted without oversight.

A case study on one round of the loop. We ran one round of this loop for a single action of the person, in an assistance setting. Before the person acts, the assistant checks whether the person can begin the action and what blocks it. If it finds a blocker, it revises the plan by inserting a help action before the person's step, following the just-right assistance policy of Liu et al. (2026), extended from reach to all three channels.

The help action takes the object occupying a hand, or brings the target within reach. We compare these decisions with decisions the same model makes directly, without self-critique, given the same policy.

Decisions with self-critique come from the BD answers in Table 2, and direct decisions come from a new run on the same questions. Both are scored against the policy applied to the reference blockers. A full loop would repeat this round after each revision and over every step of a plan, using the SV and PS checks above.

282 BD questions (no movement or posture blockers)Fully correct ↑Over-assistance ↓Under-assistance ↓Wrong type ↓
Claude Opus 5.5, direct55.7%18.6%16.1%43.5%
Claude Opus 5.5, self-critique62.4% (+6.7)20.0% (+1.4)11.8% (−4.3)35.8% (−7.7)
GPT-5.6 Terra, direct45.4%18.6%29.2%52.0%
GPT-5.6 Terra, self-critique56.0% (+10.6)14.3% (−4.3)25.0% (−4.2)38.4% (−13.7)
Qwen3.8-27B, direct44.7%11.4%28.9%57.3%
Qwen3.8-27B, self-critique47.5% (+2.8)42.9% (+31.4)13.2% (−15.7)48.9% (−8.4)
Qwen3-VL-32B-Instruct, direct28.4%7.1%73.1%73.7%
Qwen3-VL-32B-Instruct, self-critique31.6% (+3.2)71.4% (+64.3)6.1% (−67.0)65.3% (−8.4)
Qwen3-VL-8B-Instruct, direct26.2%5.7%86.3%72.4%
Qwen3-VL-8B-Instruct, self-critique36.9% (+10.6)21.4% (+15.7)42.0% (−44.3)60.2% (−12.3)
Always help27.3%100%0%63.7%
Never help24.8%0%100%–

One round of self-critique improves the help decisions, and how much depends on how reliably the model diagnoses blockers. It raises fully correct decisions for all five models, significantly for Claude Opus 5.5 (p = 0.02), GPT-5.6 Terra (p < 0.001), and Qwen3-VL-8B-Instruct (p < 0.001) in paired exact McNemar tests. For the two strongest models, it improves assistance overall, as under-assistance and wrong-type help fall while over-assistance changes little. For the open models, over-assistance also rises, because their diagnoses report blockers that are not there. Qwen3-VL-32B-Instruct, for example, rarely helps when deciding directly (78.0% "no help"), but its diagnoses report a blocker in 50 of the 70 questions that need no help. BodyReady's BD task measures this reliability, with 65.7% and 50.6% for the two strongest models against 26.2% for Qwen3-VL-32B-Instruct. For the strongest models, the remaining limit is reading the person, as Claude Opus 5.5 decides correctly in 87.9% of cases with the verified state.

Our results suggest that BodyReady's tasks could serve as the checks of such a loop, and could indicate for which models the loop is likely to help. We therefore see test-time self-critique and replanning as a promising use of BodyReady, and thank you for suggesting it. The revision adds a paragraph relating the tasks to this loop.

AuthorsOpenReview ↗

Response to Reviewer Aht7 (3/4)

W3: Data collection setups

Thank you for these points. We chose Ego-Exo4D and Nymeria as sources because both extend directly to video and to other camera views.

Static frames and temporal aspects. BodyReady evaluates readiness at a given moment, that is, whether this person can begin an action now. Temporal aspects such as pace and duration are beyond its current scope, and the framework can be extended to them with the source videos. Every annotated frame has a timestamp in its source video, and each action's completion effect specifies which channel changes when the action is done, for example the hand becoming free after placing an object. Annotating when each such change happens in the video would make readiness a function of time, and would support questions such as how long a blocker persists, when the person becomes ready for an action, or how quickly the person completes a sequence of steps. The current questions and reference answers already apply to video clips that end at the annotated moment, without re-annotation; if the reviewers find it useful, we would be glad to release these clips together with the images. The revised Limitations section states this scope.

Camera angles. Both datasets provide time-synchronized, calibrated cameras. Ego-Exo4D records each scene with 4–5 exocentric cameras and an egocentric camera, and Nymeria adds the wearer's egocentric view to the observer's view. A person's bodily state at an instant does not depend on the viewpoint, so the annotations and reference answers carry over to the other views; only the visibility of the relevant body parts and objects needs to be re-checked, so the extension is straightforward.

Bounding boxes rather than masks. We agree that BodyReady does not annotate segmentation masks or object attributes such as shape, material, and temperature. At the same time, the labels do not rely on the bounding box alone. Whenever an action is bound to an object, human reviewers check whether that action applies to this object in its current state and scene (object-level eligibility, Section 3.1 and Appendix C; Figure 2). An action enters the benchmark only if it passes this check. For example, “close the cabinet” is not posed for a cabinet that is not open, and “pick up” is not posed for an object that offers no part a person could grasp. In this way, object properties that decide whether an action applies can enter the labels in the current setting. The revised Limitations section states this scope explicitly, noting that the benchmark has no segmentation masks or attribute labels.

Q1: Scene and task complexity

Yes. The revised appendix adds the following statistics, for the scenes and for the task instructions.

ScenesMeanMedianRange
Object density: annotated objects per image and person11.2101–39
Distractors: annotated objects not mentioned in the question10.180–38
Same-category distractors (e.g., a second cup)1.200–16
Image area occupied by the designated person13.8%11.6%2.0–37.9%
Task instructionsMeanMedianRange
Prompt length (words)280287219–379
Objects mentioned per question1.810–8
Candidate actions per question: AS4.142–5
Candidate actions per question: PS2.832–5
Steps per sequence: SV2.632–3

Object counts cover annotated objects only, so they are lower bounds on scene content. The appendix reports the same statistics separately for Ego-Exo4D and Nymeria.

Reviewer_GVtrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BodyReady evaluates whether a specific person’s visible bodily state permits an action to begin and how preceding actions change subsequent applicability. It represents bodily state through hand availability and object control, posture, and target reachability. Reference answers are derived from human-reviewed visual annotations, fixed action prerequisites, and bounded completion effects. The benchmark comprises 1,190 questions over 280 images, covering 42 action templates and six tasks, including readiness assessment, blocker diagnosis, preparation selection, and sequence validation. Evaluation of 18 VLMs reveals incomplete blocker diagnoses, invalid preparation choices, and errors when validating reordered action sequences. The main contribution is a focused evaluation resource for person-specific visual action applicability.

Critical strengths
  1. A focused evaluation target with practical relevance to human assistance.

The paper clearly distinguishes an action that is plausible in context from one that a particular person can begin in their current bodily state (§3). Hand availability, posture, and reachability provide a concrete way to examine this distinction. Connecting readiness judgments to blocker diagnosis and preparation selection makes the benchmark relevant to assistance: recommending a sensible action is insufficient if the person cannot initiate it. The main value lies in this focused evaluation perspective and its dataset, rather than a new planning algorithm.

  1. The blocker analysis reveals a useful distinction between correct decisions and correct explanations.

The conditional analysis asks whether models identify the correct blocker categories even when they correctly judge an action to be unready (§5.1). The strongest model still gives an incorrect blocker set in 28.88% of those cases. The breakdown into omitted and spuriously added hand, posture, and reachability constraints makes this result more informative than a single accuracy score. This is consequential because a correct rejection alone does not identify what preparation would actually enable the action.

Critical weaknesses
  1. Limited conceptual novelty beyond a domain-specific instantiation of established action reasoning.

The paper’s contribution is limited in terms of novelty. Its formulation applies established state, precondition, and effect-based reasoning to a compact representation of human bodily conditions. The six tasks largely correspond to different queries over this shared representation, including applicability checking, identifying unmet prerequisites, and validating state transitions. Focusing on a particular observed person is useful, but the resulting contribution is primarily a specialized benchmark rather than a substantial conceptual or methodological advance. The reported failures illustrate difficulties in satisfying these constraints, but offer limited new insight into the underlying reasoning problem. My concern is therefore with the scale of the intellectual contribution, rather than the absence of a new model or additional validation experiments.

  1. The dynamic tasks conflate reasoning errors with disagreement about an unstated action model.

Preparation Selection and Sequence Validation are scored using fixed completion effects that are not included in the evaluated prompts (Appendices A, D, and E). For example, opening or closing storage is defined to leave the operating hand available. However, successfully opening a drawer does not necessarily imply releasing its handle. A model that retains hand contact can therefore derive a different subsequent readiness judgment without making a state-tracking error.

This is a problem with the task definition: the reference answer depends on an endpoint convention that is neither visible in the initial image nor uniquely implied by the action description. The paper explicitly acknowledges this issue, but its consequences remain central to the evaluation. Some reported preparation and sequence failures can reflect disagreement with the benchmark’s hidden transition rules rather than a failure to reason about the observed person. This further weakens the diagnostic value of the benchmark, which is the paper’s main contribution.

Questions for authors
  1. Beyond applying state, precondition, and effect-based reasoning to person-specific bodily conditions, what is the principal conceptual advance? Which result provides a new insight into action reasoning, rather than documenting errors on this particular task set?

  2. What in the evaluated action wording determines the reference completion state, such as release of the handle after opening a drawer? If the wording does not determine it, how are alternative physically plausible endpoints treated in scoring?

6 comments
AuthorsOpenReview ↗

Response to Reviewer GVtr (1/3)

Thank you for the thoughtful review. We appreciate your recognition that the paper distinguishes an action that is plausible in context from one that a particular person can begin, and that correct decisions and correct explanations can come apart.

W1 / Q1: Principal conceptual advance and new insight

Thank you for this question. We agree that BodyReady applies standard precondition and effect reasoning, and we chose this so that every answer can be derived from the person's state and checked. What it adds is the ability to test two assumptions that are implicit in current evaluations of action reasoning. Our results show that both fail for a person observed in an image.

  1. Good performance when the person's state is given as text does not establish reliable reasoning about an observed person.
  2. Correctly judging that a person cannot begin an action does not establish knowing what help the person needs.

These distinctions matter because a model that assists a real person faces both gaps. It is not given the person's state and must read it from what it observes, and it must choose help that fits why the person cannot act. Among the closest existing benchmarks, those that give the state as text, such as ACPBench (Kokel et al., 2025), cannot reveal the first gap, and those that score only the chosen action, such as EgoPlan-Bench (Chen et al., 2026), cannot reveal the second. Their scores can therefore overstate how ready a model is to assist a real person.

1. A state given as text versus an observed person. Action reasoning benchmarks such as ACPBench (Kokel et al., 2025) describe the state in text and ask, for example, whether an action applies. BodyReady asks such questions about a person in an image, and each image also has a human-verified state. We can therefore pose every question twice, once from the image and once with the verified state given as text in place of the image.

All 1,190 questionsAccuracy from the imageAccuracy with the verified state
Claude Opus 5.575.091.8
GPT-5.6 Terra62.881.5
Qwen3.8-27B52.966.1
Qwen3-VL-32B-Instruct42.262.9
Qwen3-VL-8B-Instruct39.349.2

With the state given as text, Claude Opus 5.5 answers 91.8% of the questions correctly. From the image of the same person, it answers 75.0%. Accuracy with the state given as text therefore overstates how reliably a model reasons about a person it observes. The open models fail even with the state given as text (Qwen3-VL-8B-Instruct, 49.2%), so for them the reasoning itself is also a bottleneck.

One recurring error shows what models miss when they must read the person themselves. A hand that only rests on something is still free to use, and every question says so ("Releasable touch can participate"). Models nonetheless often treat such a hand as occupied. In the 74 BD questions where a hand rests on something and no hand blocks the action, Claude Opus 5.5 reports a hand blocker in 24 from the image but in only 4 with the verified state, and GPT-5.6 Terra in 21 and 4, respectively. A probe that asks only for each hand's state, with no action to reason about, shows that the hand itself is misread, since Claude Opus 5.5 identifies a resting hand correctly in only 14 of 48 cases. A benchmark that gives the state as text cannot reveal this error, which is specific to reading a human body.

AuthorsOpenReview ↗

Response to Reviewer GVtr (2/3)

2. Judging that a person cannot act versus knowing how to help. Benchmarks that score only the chosen action, such as EgoPlan-Bench (Chen et al., 2026), cannot tell whether a model knows why the person cannot act. BodyReady also scores the reason. Claude Opus 5.5 gives an incomplete reason in 28.88% of its correct BD rejections, the result you highlight. The reason matters because it decides the help. A hand holding an object calls for taking the object, and an out-of-reach target calls for bringing it closer.

To test whether this matters for assistance, we turned each model's diagnosis into a help decision with the just-right assistance policy of Liu et al. (2026), extended from reach to all three channels. We compared these decisions with decisions the same model makes directly, given the same policy, on 282 of 344 BD questions (excluding blockers that need movement or a posture change). All decisions are scored against the policy applied to the reference blockers.

Fully correct (%) ↑DirectImage diagnosis + policyVerified-state diagnosis + policy
Claude Opus 5.555.762.487.9
GPT-5.6 Terra45.456.077.0
Qwen3.8-27B44.747.566.0
Qwen3-VL-32B-Instruct28.431.652.5
Qwen3-VL-8B-Instruct26.236.937.9

↑ means higher is better. For reference, always helping is fully correct in 27.3% of cases, and never helping in 24.8%. Diagnoses from the image are the BD answers reported in Table 2, and direct decisions come from a new run on the same questions.

Knowing that help is needed does not mean knowing which help. Among the cases in which the person needs help and Claude Opus 5.5 gives it, the help is of the wrong type in 43.5% when it decides directly. Deriving the help from the model's own diagnosis raises fully correct decisions for all five models, significantly for Claude Opus 5.5 (p = 0.02), GPT-5.6 Terra (p < 0.001), and Qwen3-VL-8B-Instruct (p < 0.001) in paired exact McNemar tests, and lowers wrong-type help for Claude Opus 5.5 to 35.8%. For the two proprietary models, assistance improves overall. For the open models, over-assistance also rises, because their diagnoses report blockers that are not there. Qwen3-VL-32B-Instruct, for example, over-assists in 71.4% of the cases that need no help, against 7.1% when deciding directly (details in our reply to Reviewer Aht7). Good help therefore depends on a correct diagnosis, and a benchmark that scores only the chosen action cannot measure it.

What we demonstrate and what we propose. In the above case study, the model still diagnoses the blocker itself, and only the choice of help is made outside the model by a fixed policy. These results motivate a potential design, which we leave to future work, in which the model only reads the person's state, and explicit rules decide everything else outside the model. This kind of design is an established direction in planning, as in LLM+P (Liu et al., 2023) and LLM-Modulo frameworks (Kambhampati et al., 2024). Yet these works address domains described in text. BodyReady offers an entry point for extending them to a person observed in an image. Its prerequisites and effects (Appendices C and D) can serve as the explicit rules, and its human-verified states can test whether the model reads the person correctly.

The six tasks. We agree that the six tasks are queries over a shared representation, and we designed them this way on purpose. The shared representation makes failures comparable across tasks, and each task brings out a failure that the others do not.

TaskWhat it exposes
ARAn action plausible in the scene that this person cannot begin now
ASReadiness judged across several candidate actions at once
BDA correct verdict with an incomplete reason
RRA judgment that must change when the person's state is edited
PSA preparation that would help but cannot be begun
SVA judgment that changes when only the order of steps changes

For example, Claude Opus 5.5 judges both orders of the same steps correctly in only 21 of 31 SV pairs, a failure that no single-state task would reveal.

In summary, BodyReady provides more than another set of model errors. Its findings point to two requirements for evaluating reasoning about a person the model observes. The model should read the person's state itself rather than receive it as text, and its reason should be scored, not only its verdict. ACPBench gives the state as text, and EgoPlan-Bench scores only the chosen action, whereas BodyReady meets both requirements.

AuthorsOpenReview ↗

Response to Reviewer GVtr (3/3)

W2 / Q2: Completion endpoints

Thank you for this question. We agree that "open the drawer" does not say whether the handle is released, and the same question applies to every completion effect. We therefore checked all completion effects in the 358 PS/SV questions, not only storage. The endpoint changes the reference answer in 2 of them and leaves the benchmark's diagnostics essentially unchanged.

What determines the reference completion state. Each action template has a fixed completion effect in the action catalog (Appendix D); for opening or closing storage, the operating hand becomes available. The action wording alone does not determine it, and answers are scored against this reference endpoint.

Check 1: an audit of all PS/SV questions. We audited every PS/SV question for whether its answer depends on the endpoint:

Unambiguous endpointAmbiguous endpoint, same answerAmbiguous endpoint, answer may change
323332

Check 2: recomputing the answers. For the two kinds of actions where the endpoint is most open to interpretation, opening or closing storage and using a held object, we recomputed all 358 reference answers under each alternative endpoint:

Alternative endpointQuestions where it appliesAnswers that changeAnswers that cannot be determined
Hand rests on the storage2900
Hand keeps gripping the storage2901
Hand keeps holding a small box420
Hand releases a held tool or container100

The first row includes your drawer example: a hand that stays on the handle is in releasable contact, which the prompt counts as available ("Releasable touch can participate"), so it gives the same answer as a released hand. In the one undetermined case, the answer depends on which hand opened the drawer, which the image does not show. The 2 changes are the same 2 questions flagged in Check 1, where the person may keep holding a small box after opening it.

We have corrected these 3 questions in the revised paper, replacing each with a reviewed question whose steps have unambiguous endpoints. Because only 3 of the 1,190 questions change, each model’s overall accuracy can change by at most 0.25 points, and all conclusions in the paper still hold. All 18 models were re-run on the new questions, and Table 2 of the revised paper reports the corrected results (Appendix J).

What this means for the diagnostics. The endpoint leaves the reported results essentially unchanged:

ResultAffected by the endpoint
PS answers0 of 80
SV answersat most 3 of 278
SV errors of the best model on SV (GPT-6 Astra)at most 3 of 53
Any model's overall accuracyat most 0.25 points
AR, AS, BD, RR, including the 28.88% you notednot involved

Even for the best model on SV, at least 50 of its 53 SV errors occur whatever endpoint is assumed. The PS and SV failures we report therefore do not depend on which endpoint is used in scoring.

References

  • Liu, Ziang and Yan, Yunting and Cheung, Christy and Ying, Tailai and Liu, Bodong and Tong, Shiqin and Orkwis, Alexander and Dimitropoulou, Katherine and Bhattacharjee, Tapomayukh (2026). Knowing When Not to Help: Active Estimation of Human Reachability for Just-Right Robot Assistance. RSS 2026.
  • Liu, B., Jiang, Y., Zhang, X., Liu, Q., Zhang, S., Biswas, J., & Stone, P. (2023). Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477.
  • Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., ... & Murthy, A. (2024). Llms can't plan, but can help planning in llm-modulo frameworks. arXiv preprint arXiv:2402.01817.
  • Kokel, H., Katz, M., Srinivas, K., & Sohrabi, S. (2025, April). Acpbench: Reasoning about action, change, and planning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 25, pp. 26559-26568).
  • Chen, Y., Ge, Y., Ge, Y., Ding, M., Li, B., Wang, R., ... & Liu, X. (2026). Egoplan-bench: Benchmarking multimodal large language models for human-level planning. International Journal of Computer Vision, 134(3), 118.
Reviewer_LEbYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BodyReady tests whether a person can begin an action from their visible bodily state. It tracks hand use, posture, and reachability. Six tasks cover current readiness, blockers, preparation, and short action sequences. The benchmark has 1,190 questions from 280 images and 42 action templates. Across 18 VLMs, the best strict accuracy is 74.96%.

Critical strengths
  1. Useful task design. The benchmark focuses on a specific person's current ability to begin an action. This is relevant to human assistance. Explicit action boundaries, separate object checks, and uncertainty filtering support a coherent evaluation (Sections 3.1-3.4). Human reviewers also made substantial corrections to automatic proposals.

  2. Informative analysis. The diagnostics reveal errors hidden by overall accuracy. For example, Opus solves 47/47 preparation questions with a valid option, but only 20/33 with none (Table 12). The paper reports conditional denominators, checks formatting effects, and uses source-group bootstrap intervals. It avoids overstating the top models' ranking.

Critical weaknesses
  1. Unstated action effects complicate the dynamic tasks. PS/SV prompts omit detailed completion conventions (Appendix E, p.19). Labels still depend on them. For example, the reference rules make the operating hand available after opening or closing storage (Appendix D, p.18). A model may infer a different endpoint and be marked wrong. The authors acknowledge this ambiguity, but do not quantify its effect. Testing explicit conventions would help separate semantic disagreement from reasoning errors. This concern mainly affects dynamic judgments.

  2. Final-answer reliability lacks independent validation. The two reviewers split the annotation pool and discussed uncertain cases (Section 3.3, p.5). This does not establish independent agreement. Final answers were derived from frozen annotations and rules (Appendix D, p.18). No human baseline or final-answer error audit is reported. Manual corrections and uncertainty filtering help, but do not measure residual errors. An independent sample is needed to assess hand-control and reachability judgments. This is a validation gap, not evidence that the labels are wrong.

Questions for authors
  1. How does PS/SV performance change when relevant action boundaries and completion effects are stated explicitly? Which errors remain?

  2. Was any final-question subset independently answered? Please report its size, agreement, and correction rate. A human baseline should use the same images and wording as the models.

  3. Can you compare image, no-image, and verified textual-state inputs on a focused subset? Keep action conventions fixed. This would help distinguish visual grounding errors from later reasoning errors.

5 comments
AuthorsOpenReview ↗

Response to Reviewer LEbY (1/3)

Thank you for the detailed and constructive review. Your three questions led to the most informative new analyses in the revision.

W1 / Q1: Unstated completion effects

Thank you for raising this. We agree that the prompts do not state completion effects such as the operating hand becoming available after opening or closing storage (Appendix D), and that a model may infer a different endpoint. We therefore quantified the effect for all completion effects in the 358 PS/SV questions. The endpoint changes the reference answer in 2 of them and leaves the benchmark's diagnostics essentially unchanged.

Check 1: an audit of all PS/SV questions. We audited every PS/SV question for whether its answer depends on the endpoint:

Unambiguous endpointAmbiguous endpoint, same answerAmbiguous endpoint, answer may change
323332

Check 2: recomputing the answers. For the two kinds of actions where the endpoint is most open to interpretation, opening or closing storage and using a held object, we recomputed all 358 reference answers under each alternative endpoint:

Alternative endpointApplies toAnswers changedUndetermined
Hand rests on the storage2900
Hand keeps gripping the storage2901
Hand keeps holding a small box420
Hand releases a held tool or container100

The first row covers your example: a hand that stays on the storage is in releasable contact, which the prompt counts as available ("Releasable touch can participate"), so it gives the same answer as a released hand. In the one undetermined case, the answer depends on which hand opened the drawer, which the image does not show. The 2 changes are the same 2 questions flagged in Check 1, where the person may keep holding a small box after opening it.

We have corrected these 3 questions in the revised paper. Each is replaced with a reviewed question whose steps have unambiguous endpoints. With 3 further corrections described under W2, including one SV question, 6 of the 1,190 questions change. Re-running all 18 models on them changes each model’s overall accuracy by at most 0.17 points, and all conclusions in the paper still hold. The revised Table 2 and Appendix J report the corrected results.

What this means for the diagnostics.

ResultAffected by the endpoint
PS answers0 of 80
SV answersat most 3 of 278
SV errors of the best model on SV (GPT-6 Astra)at most 3 of 53
Any model's overall accuracyat most 0.25 points
AR, AS, BD, RRnot involved

Semantic disagreement about endpoints therefore accounts for at most 3 of the 278 SV answers, and for none of the PS answers.

Q1: stating the action boundaries and completion effects explicitly. We added to the original prompt the state each action leaves when completed (Appendix D), for every PS candidate and every SV step before the last. The rest of the prompt is unchanged.

The table gives accuracy with the original prompts and with the action boundaries and completion effects stated. For the open models, the original-prompt column is taken from Table 2 of the original submission. Because repeated runs of the proprietary models vary by a few points, we re-ran their original prompts in the same run as the new condition.

ModelPSSV
Claude Opus 5.580.0 → 82.580.6 → 80.2
GPT-5.6 Terra68.8 → 67.567.6 → 55.8
Qwen3.8-27B53.8 → 45.052.9 → 49.3
Qwen3-VL-32B-Instruct40.0 → 31.343.9 → 56.1
Qwen3-VL-8B-Instruct36.3 → 28.832.0 → 30.6

Stating the action boundaries and completion effects changes some answers, but substantial errors remain. Qwen3-VL-32B-Instruct gains 12.2 points on SV, almost entirely on questions whose answer depends on a state change (+35 of 137, against −1 of 141 on the others). The strongest model is essentially unchanged, and most other models score the same or lower. GPT-5.6 Terra, for example, answers 33 fewer SV questions correctly, with 8 more outputs cut off by the budget and 25 more wrong answers.

On PS, the open models also change which wrong answers they give. With each candidate's effect written out, Qwen3.8-27B and Qwen3-VL-32B-Instruct more often select candidates that would make the target ready but that the person cannot begin (12 → 20 and 3 → 13 of 31 such candidates), while Qwen3-VL-8B-Instruct more often selects nothing.

Which errors remain. On the 40 SV questions that people also answered (W2), people score 100% and 97.5%, while the models remain well below.

Same 40 SV questionsAccuracy with action boundaries and completion effects
Claude Opus 5.577.5
GPT-5.6 Terra55.0
Qwen3.8-27B45.0
Qwen3-VL-32B-Instruct52.5
Qwen3-VL-8B-Instruct32.5

Q3 shows where these errors arise, in reading the person or in applying the action requirements.

AuthorsOpenReview ↗

Response to Reviewer LEbY (2/3)

W2 / Q2: Independent validation and a human baseline

Thank you for this suggestion. We validated the benchmark at two levels: the final answers, through an independently answered subset (size, agreement, and correction rate below), and the state judgments behind them, through independent double annotation.

1. Human baseline. Two people outside the project answered 200 questions stratified by task (AR 30, AS 30, BD 40, RR 30, PS 30, SV 40). They saw the same images and prompts as the models and submitted all answers before seeing any reference answer. Agreement with the reference is strict accuracy (%). All scores, for people and models, use the reference answers after the correction described below.

Same 200 questions, original promptsAR (30)AS (30)BD (40)RR (30)PS (30)SV (40)All (200)
Person 110083.310010010010097.5
Person 210093.385.010010097.595.5
Claude Opus 5.586.763.365.083.393.380.078.0
GPT-5.6 Terra86.746.755.073.376.767.567.0
Qwen3.8-27B83.330.045.063.353.350.053.5
Qwen3-VL-32B-Instruct83.336.727.550.043.335.044.5
Qwen3-VL-8B-Instruct73.320.042.550.040.035.043.0

People's errors concentrate in AS and BD (13 of their 14 errors), the same two tasks on which all six proprietary models score lowest.

Correction rate: 1 of 200. We checked each disagreement between a person and the reference against the written definitions in the prompt. One was a reference error. An AS question listed as executable an action that had already been completed in the scene. All other reference answers were confirmed. Following your suggestion, we also checked every other question built on the same scene facts and found two more that use the completed action, one BD and one SV, both outside the 200.

The revised benchmark removes the action from the AS question and replaces the other two questions. Together with the three endpoint questions above, six questions change. Re-running all 18 models on them changes each model's overall accuracy by at most 2 questions (0.17 points), and all conclusions in the paper still hold.

2. Agreement on state judgments. Two annotators independently annotated 100 additional images from the same source datasets, from scratch, without automatic drafts and without discussion. Agreement is computed over the items that both annotators labeled (excluding "unknown"):

ChannelItems comparedAgreement
Hand control (available, or holding or controlling which object)156 hands90.4%
Posture100 persons98.0%
Reachability794 person–object pairs90.7%

3. A reachability check that does not rely on human annotation. On the Nymeria portion of the benchmark, reachability labels agree with 3D hand–target distance (AUROC 0.91, 95% CI 0.88–0.95; details in our reply to Reviewer Aht7).

AuthorsOpenReview ↗

Response to Reviewer LEbY (3/3)

Q3: Image, no-image, and verified textual-state inputs

Thank you for this suggestion. We ran the three inputs on all 1,190 questions with the original prompts, so the action conventions are the same in all three, and the image results are those in Table 2 of the original submission. The verified textual state is given in place of the image. It lists the designated person's human-verified hand states, posture, and reachability of each relevant object; it gives no answers, no blockers, and no states after an action. All three inputs use the same decoding settings. Repeated runs of the proprietary models vary by a few points (Claude Opus 5.5 scores 83.8 on PS in Table 2 and 80.0 in the re-run of Q1), far less than the differences between the three inputs.

All 1,190 questionsNo imageImageVerified stateImage errors corrected by the verified stateErrors under both inputs
Claude Opus 5.543.375.091.822672
GPT-5.6 Terra39.262.881.5325118
Qwen3.8-27B35.552.966.1257304
Qwen3-VL-32B-Instruct35.442.262.9341347
Qwen3-VL-8B-Instruct32.939.349.2239483

Each error made from the image falls into one of the last two columns. It is either corrected when the verified state is given, or it remains under both inputs.

Without the image, accuracy falls for every model, so the answers depend on the image. For the proprietary models, most errors made from the image are corrected by the verified state (Claude Opus 5.5, 226 corrected against 72 that remain). This suggests that their main difficulty lies in reading the person from the image, which bears on the visual grounding errors you ask about. One recurring error of this kind is described in our reply to Reviewer GVtr. For the open models, as many or more errors remain under both inputs (Qwen3-VL-8B-Instruct, 483 remain against 239 corrected). This suggests that they also struggle to apply the action requirements to a given state, which corresponds to the later reasoning errors you mention.

The revised paper adds this comparison to the appendix. This addresses a limitation noted in Appendix A of the submission.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their careful reading and constructive suggestions. We are encouraged that the reviewers see value in the following aspects of the work.

  • The focus on whether a specific person can begin an action now, with practical relevance to human assistance (LEbY, GVtr).
  • A clear formulation and useful task design (Aht7, LEbY).
  • Informative diagnostics beyond overall accuracy, including the distinction between correct decisions and correct explanations (LEbY, GVtr, Aht7).

We answer each reviewer's questions in the individual replies. In response to them, we ran new analyses and independent validation studies, and the revised paper adds the following.

Prompted by Reviewer LEbY

  • Image, no-image, and verified-state inputs on all 1,190 questions (appendix).
  • Results with the action boundaries and completion effects stated in the prompt (appendix).
  • Human baseline on 200 questions with its correction rate, and independent double annotation (appendix).

Prompted by Reviewer GVtr

  • Audit and recomputation of all PS/SV answers under alternative endpoints (appendix).
  • What the results imply for evaluating action reasoning, with the resting-hand analysis and the help case study (main text).
  • Related work on planning with external reasoners.

Prompted by Reviewer Aht7

  • BodyReady's tasks as checks in test-time self-critique and replanning, with a one-round case study (main text).
  • Extension of the formulation beyond human bodies (discussion).
  • Scope regarding temporal aspects such as pace and duration, and the lack of segmentation masks and attribute labels (limitations).
  • Scene and instruction complexity, and a 3D check of reachability labels (appendix).

The human baseline and the endpoint audit also led us to correct 6 of the 1,190 questions. Re-running all 18 models on them changes each model's number of correct answers by at most 2 of 1,190 (0.17 points), and all conclusions in the paper still hold. The revised Table 2 reports the corrected results (Appendix J).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.