acceptodds
Under review as a conference paper at ICLR 2027

Modeling the object representations underlying human physical reasoning

Abstract

Humans appear to represent objects when reasoning about physics with coarse, volumetric "bodies" that smooth concavities, trading fine visual detail for efficient physical predictions. Yet, the structure of these representations remains largely unknown. Segmentation models, in contrast, are trained for pixel-accurate masks that may misalign with such bodies. We ask whether and when these models nonetheless acquire human-like object representations. Using a time-to-collision (TTC) and change detection (CD) behavioral task with data from 226 and 50 human participants, respectively, we introduce a pipeline and an alignment metric to compare the visual representations of segmentation models to those of humans. We do this systematically on multiple architectures (DINOv2, SegFormer, DeepLabV3+, and UPerNet), varying their size and training time. We find that briefly trained models segment objects too coarsely, aligning poorly with humans, while fully trained models segment objects too finely. For each model, there is an intermediate training regime that best matches the coarse bodies observed in human behaviour, and larger models tend to reach it earlier. We show these bodies emerge under resource constraints in general-purpose vision models, providing computational support to resource-rational accounts of human cognition. This work provides a foundational framework for testing alignment between vision models and humans and shows there is a growing gap between the state-of-the-art in artificial intelligence and human cognition, driven by scaling model size and training.

Primary area:
applications to neuroscience & cognitive science
Keywords:
computer vision, segmentation models, alignment, human-model alignment, object representations, training dynamics, cognitive science, resource rationality

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wAPwOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at whether segmentation models can generate rough object masks and thereby make human-like judgments in physical reasoning tasks involving concave shapes.

Critical strengths

It's a concrete, well-controlled behavioral alignment pipeline. It turns model masks into predictions for the exact same TTC and change-detection stimuli that humans saw, and it also feeds in the true velocity so you can isolate how much of the effect comes from the representation itself. So it's testing a precise geometric hypothesis rather than relying on generic segmentation metrics or eyeballing the masks.

Critical weaknesses

I just have one small concern: Section 5's conclusion interprets the intermediate checkpoints and the smaller or less-trained models as evidence that resource limits give rise to human-like bodies, and that scaling effects lead to a growing representational gap between humans and machines. But the actual intervention is really just evaluating standard segmentation learners at different training steps and capacities — it never explicitly sets up a trade-off between physical prediction quality and quantified resource cost, complexity measures, or compute budget.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer wAPw: Clarifying Resource Rationality and Scope

We thank Reviewer wAPw for their positive review and insightful comment. We incorporate the suggestion into our work as follows:

We have removed the resource rationality claims from the abstract and conclusion, and revised Section 5 to present it as a possible interpretation, distinct from our empirical findings. We have also narrowed our claims about human-model divergence and scaling to the specific behavioral effects studied here.

These changes in sections 5-6 are marked in magenta in the revised PDF. We sincerely thank Reviewer wAPw for their positive review and hope to have addressed their main concern. We remain available for any further clarifications.

Thank you for your reply, my main concern has been solved. Because the maximum score is 4, I cannot increase my score any further. I believe this is a very good piece of work and deserves to be accepted at ICLR.

Thank you for your support of our work and for the thoughtful review.

Reviewer_8WuBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether segmentation models develop object representations whose behavioral consequences resemble human physical judgments. The authors introduce an evaluation pipeline that converts predicted segmentation masks into time-to-collision (TTC) predictions using ground-truth object velocities. Model–human alignment is primarily measured by agreement in the average TTC difference between convex and concave conditions. Across four architecture families and multiple model sizes, this difference generally approaches the human value and subsequently moves away from it as training progresses. The authors also report change-detection experiments in which models, like humans, are less sensitive to changes within concavities than to changes on convex regions. They interpret these findings as evidence that intermediate training stages produce object representations resembling the coarse approximations underlying human physical reasoning.

Critical strengths
  1. Systematic characterization across models and training stages. The study examines four architecture families, 16 configurations, and multiple training seeds, and evaluates checkpoint selection using held-out participants. This breadth supports the descriptive finding that intermediate training stages can match the average human concavity effect more closely than early or late checkpoints.

  2. Complementary evidence from change detection. The change-detection experiment provides evidence beyond the TTC comparison: models reproduce the direction of the human asymmetry between concave and convex changes. Together with the mask visualizations, this helps connect the observed behavioral effects to the geometry of the predicted shapes.

Critical weaknesses
  1. The additional insight beyond a generic coarse-to-fine learning process is insufficiently established. Prior work already provides evidence for coarse human object representations and concavity filling (Li et al., 2023), as well as neural networks’ tendency to learn simpler, low-frequency structure before finer detail (Rahaman et al., 2019). Both are acknowledged in the paper. Together, these findings suggest a straightforward explanation for the reported trajectory: early masks overfill concavities, continued training resolves their boundaries, and the resulting TTC difference passes through the human value. Although this does not make the precise empirical results inevitable, it limits how surprising or explanatory an intermediate minimum in alignment error is. The geometric operators in Section 4.1 and Appendix D.5 also match the human effect closely, but only their fitted settings are reported. Showing how alignment changes as their parameters progressively refine the shapes would help determine whether the model trajectory reveals anything beyond changing geometric coarseness. The observation that this geometry arises during segmentation training is supported, but its additional significance for understanding human representations remains unclear.

  2. The behavioral comparisons do not sufficiently substantiate the broader claim of human-like representations. The TTC metric matches a difference between conditions, averaged over collision times; it does not require accurate predictions in either condition separately. The authors appropriately acknowledge this limitation in Section 3.1, but do not report the condition-specific human–model comparisons needed to assess its consequences. Stronger evidence would show that convex and concave TTC predictions each match human judgments, or that their discrepancies are consistent with a common response-time offset that generalizes across models and stimuli. Such an offset could be consistent with a human response delay, although it would not establish that explanation by itself. Furthermore, although the stimuli include multiple shapes and change sizes, the aggregate TTC and change-detection results do not establish whether models reproduce how human judgments vary with the type and degree of concavity or convexity. These missing comparisons limit the interpretation of the selected checkpoints as models of human object representations.

Questions for authors
  1. Specificity of the alignment trajectory. Can the authors report alignment curves obtained by systematically varying the coarseness parameters of the geometric operators in Appendix D.5? Which aspects of the model trajectories provide evidence beyond the progression from coarse to increasingly precise shape representations?

  2. Condition-specific TTC agreement. At the selected checkpoints, how closely do model TTC predictions match human judgments separately in the convex and concave conditions, across stimuli and collision times? If a response-time offset is needed, can a single offset explain both conditions and remain consistent across models, rather than being fitted separately to each model’s aggregate results?

  3. Dependence on shape geometry. Can the existing data be analyzed by shape and change magnitude to determine whether model and human judgments vary together? More specifically, what evidence supports agreement in sensitivity to the type and degree of concavity or convexity, beyond the average difference between the two conditions?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8WuB: Additional Analyses and Behavioral Agreement

We thank Reviewer 8WuB for the review and the thoughtful feedback. We have carried out the requested analyses, which clarify how human-like biases emerge during segmentation training and reveal agreement with human judgments beyond the average concavity effect. New relevant sections and modifications are all marked in orange in the revised version of the PDF.

1. What do segmentation models add beyond geometric operators?

Sweeping the parameters of closing, dilation, and α-shapes produces intermediate minima in alignment error, with held-out errors of 6.4-6.6 ms, comparable to most models. These baselines help characterize the geometric approximations underlying the behavioral effect (Appendix D.5).

Our contribution addresses how such approximations could arise through learning. The operators start from ground-truth masks, with their parameters selected to match human responses. They quantify which transformations reproduce the bias, without explaining how these approximations arise when learning to segment objects from images.

Our models learn segmentation from images without human responses being used to train their weights. Training itself produces intermediate states whose predictions resemble human judgments. Human responses are used to select checkpoints, but do not guide the learning process. This provides a possible account of how these approximations emerge during segmentation learning. The comparisons below further test which aspects of human behavior they reproduce.

2. Do model predictions match humans in each TTC condition?

We compared predictions separately for convex and concave conditions across all 96 videos. Mean model-human discrepancies range from −127 to +24 ms for concave trials and −128 to +11 ms for convex trials. A single offset shared across models leaves 58 ms RMSE, compared with 40 ms using model-specific offsets.

This analysis separates the concavity effect from absolute response timing. Within each model, similar offsets for the two conditions follow from matching their mean difference. Across models, however, a common response delay does not account for the discrepancies, and residual errors also vary with collision time. Our TTC comparison thus isolates agreement on the behavioral effect of concavity, while reproducing absolute human response timing remains a separate modeling question.

3. Do models reproduce differences across shapes and change sizes?

The shape-level analysis provides additional evidence for SegFormer: all six configurations track variation in the human TTC concavity effect across the eight tested shapes, with correlations of 0.79-0.88. Selecting checkpoints on the average effect does not guarantee this agreement. DINOv2 Base, for example, matches the average most closely but has a shape-level correlation of -0.34. Geometric operators also reproduce the shape profile well. These comparisons identify which segmentation models capture the pattern of human judgments beyond the aggregate result.

Change detection reveals a further distinction. In all 16 configurations, additions inside concavities produce greater mask sensitivity when they leave the concavity unfilled than when they fill it: mean relative area change is 0.89 versus 0.22. This matches the human ordering and shows that sensitivity depends on whether a change fills a concavity, beyond its location inside one. At TTC selected checkpoints with the fixed threshold, 14 of 16 configurations also reproduce the higher detection of larger concavity-filling changes. Agreement concerns these systematic contrasts: effect magnitudes differ, and models do not reliably capture variation between individual stimuli within each change type.

Together, these analyses show how segmentation training produces approximations that reproduce human behavioral biases, with SegFormer additionally capturing how the TTC effect varies across shapes.

We sincerely thank Reviewer 8WuB for their review, which helped us improve the paper, and hope to have addressed their main concerns. We remain happy to clarify any remaining points. We would be sincerely grateful if they considered increasing their score in light of the additional analyses and revisions made in response to their feedback.

Thank you for the additional experiments and for clarifying the scope of the claims. I appreciate the additional evidence from the shape-level and change-detection analyses. However, the geometric baselines also exhibit intermediate alignment minima and similarly high correlations across shapes. Furthermore, models with similar agreement on the average convex–concave TTC contrast can differ substantially in their absolute TTC predictions and shape-level agreement with humans. These results reinforce my concern about the limited diagnostic value of the aggregate contrast for comparing candidate object representations.

I acknowledge the narrower finding that these geometric approximations can emerge during segmentation learning. Nevertheless, the additional insight into human object representations beyond the existing coarse-body account remains insufficiently established. The revised claims are more appropriately scoped, but my concern about the significance of the contribution remains. I therefore maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer 8WuB: Operators, Diagnostic Value, and Significance

We sincerely thank Reviewer 8WuB for the careful reading of the new analyses. We are glad that we now agree on the findings and on their scope. Where we respectfully differ is on their significance, and we explain why below.

The operators also show intermediate minima and track the shape profile. Yes, and this is the comparison we intended. The operators are given the true mask and a parameter fitted to human responses. SegFormer, given only images and a segmentation loss, is comparable to them on both tests: the average effect (5.3 ms held-out for B0, vs. 6.4 to 6.6 ms) and the per-shape profile (r = 0.79 to 0.88, vs. 0.85 to 0.93). The operators define the target geometry; the result is that a learner reaches it without being told.

Models with the same average differ on stricter tests. This is what makes the framework useful for comparing candidate representations. The average effect is a first test that all 16 configurations pass. The per-shape profile separates them, and SegFormer passes it at all six sizes with no checkpoint selected on it. The revision now presents both levels in the main text (Sec. 4.1).

Significance. Here we respectfully disagree. The coarse-body account describes the approximation but not its origin. We show that it can arise from ordinary segmentation learning with no human supervision, in four architecture families and from random initialization, and that further training makes models more accurate and less aligned with human judgments. We believe this is an insightful finding for the neuroscience and cognitive science community at ICLR, the primary area of this submission: it gives the coarse-body account a concrete candidate for how such representations are learned, and an open pipeline to test other models against the same human data.

We would like to thank you again for your thoughtful feedbacks, and for prompting these analyses which made the paper stronger all together.

Thank you for the clarification. Many architectures and geometric operators can match the average TTC contrast, while models with similar aggregate scores still differ substantially in absolute timing and shape-level agreement. This limits what the aggregate match establishes about their underlying representations. The shape-level correlations add a behavioral constraint, but do not directly establish geometric similarity. This distinction matters for the statement that the learner “reaches” the geometry defined by the operators: a direct comparison of model and operator masks, such as IoU or boundary agreement, would address that correspondence more directly than TTC correlations. I accept that coarse masks emerge during segmentation training, but the claimed geometric correspondence and the additional insight into human object representations beyond the existing coarse-body account remain insufficiently established. I therefore maintain my original rating.

Reviewer_AgQtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that vision models trained under resource constraints (limited training time and model size) could give rise to object representations that are similar to that in humans. In particular, so trained models give an approximation representation/segmentation of objects, which seem to align with humans when inferring physics.

Critical strengths

Using modern foundation models to better probe human perception and cognition and timely and important.

Critical weaknesses
  • The wording at many places is not very precise, which makes it really hard to understand precisely what the authors are trying to say. For instance, the abstract says "Humans appear to represent objects when reasoning about physics with coarse, volumetric "bodies" that smooth concavities, trading fine visual detail for efficient physical predictions. Yet, the structure of these representations remains largely unknown." What precisely do you mean representations? Do you refer to representations of objects in the brain? What do you mean structure? Do you mean the dimensionality of the encoding space and what each dimension encodes? Also you say "he problems of ‘what’, ‘where’, and ‘how’ are coupled but separate". What do you mean coupled? What do you mean separate? They are hollow words that ideally should show up only after you provide concrete definitions. I suggest use plain english.

  • I think object representation is an interesting question, but the authors seem to equate object representation with just segmentation. Are there other aspects of object representation that might be important? Why focus only on segmentation?

  • I don't fully understand how you compare to human behaviors. I would have thought that you would do something like maximum likelihood estimation, since human judgment is noisy, both within a subject and across subjects. Could you explain more clearly how you compare with humans? Also, I didn't realize this until later; the way you say that a model's representation aligns with humans is not to actually compare with say fMRI recordings, but actually by showing that the model can better predict human responses? If so, this seems to be just a necessary but insufficient criterion for human alignment, right?

Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer AgQt: Clarifying Terminology and Behavioral Comparisons

We thank Reviewer AgQt for their review. We really appreciate the feedback and have incorporated it into the revised manuscript. We have clarified our terminology, the rationale for studying segmentation, and the scope of our behavioral comparisons. The relevant passages are highlighted in blue in the revised PDF.

1. Meaning of “representation” and “structure.”

By “object representation” here we mean the object geometry used to predict collisions or detect shape changes (TTC and CD, respectively). For models this geometry is explicitly given by segmentation masks while for humans it is inferred indirectly from their behavioral responses. By “structure” we mean geometric properties, such as how much concavities are filled, rather than the dimensionality or organization of neural encodings.

We have clarified these terms in the abstract and introduction (section 1), and replaced “coupled but separate” with a concrete explanation of why recognition and physical prediction may require different levels of detail in the shape. We have also clarified the opening of the discussion (section 5). These revisions are highlighted in blue.

2. Why segmentation?

We agree that object representations include more than shape. Segmentation provides explicit boundaries which allows us to isolate how their geometry affects predictions (for example in collisions, keeping motion fixed). This directly examines the proposed role of concavity filling in human judgments. Mass, material, texture, and other properties are outside the scope of our tests. The revised paragraph (in blue) in the introduction makes this rationale explicit.

3. Comparison with noisy human responses.

For TTC, we measure how much earlier, on average, humans report collisions in the concave condition than in the convex condition, and we then test whether models predict a similar difference. Because human judgments vary across participants, we report a 95% confidence interval for their average convex-concave difference. To test whether a model’s agreement extends beyond the participants used to choose its checkpoint, we select the checkpoint on one group and evaluate it on another, repeating this across random splits. For CD, we compare detection rates using a threshold fitted once on a reference model and then applied unchanged to the others.

Our analysis targets average behavioral effects and their uncertainty. We therefore compare these effects directly, without fitting a likelihood model of individual responses. The blue additions in the methods overview (3) and TTC methods (3.1) clarify these comparisons, uncertainty, and held-out evaluation. The CD threshold procedure is highlighted in sec. 3.2.

4. What behavioral agreement establishes.

We agree that matching human responses does not establish that humans and models use the same representations. The revised parts in blue in TTC methods (3.1) and limitations (6) now make this distinction explicit. Our contribution is to show that agreement with the average human TTC concavity effect peaks at intermediate stages of segmentation training, and that models can reproduce the human concavity filling effects.

We sincerely hope to have addressed Reviewer AgQt’s concerns, which helped us improve the paper. We are available for any clarifications and would be sincerely grateful if Reviewer AgQt would consider increasing their score.

AuthorsOpenReview ↗

Follow-up to Reviewer AgQt

As the discussion period is closing, we would like to sincerely thank Reviewer AgQt for the constructive feedback.

The revision addresses each of your points: terms are defined in simpler language (abstract, Sec. 1), the reason for focusing on segmentation is stated (Sec. 1), the human comparison is spelled out (Sec. 3, 3.1), and what behavioral agreement establishes is stated explicitly (Sec. 3.1, 6). We also added stricter tests of agreement with human judgments (per-shape agreement and a full change-detection analysis).

Thank you again for your help in making the paper clearer. We hope we have answered your concerns, and we would be very grateful if you could consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.