- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SPhyR recasts 2D topology-optimisation as a grid-completion task and grades outputs against a physics simulation rather than a stored answer, which captures what cell-matching misses. The paper releases SPhyR as an OpenEnv environment with optional simulator feedback between attempts, which helps a model to revise its answers. It evaluates eleven models zero-shot on the v1-dataset and three models on the v2 dataset, demonstrating the advantage of physics based verifer rather than cell matching.
Critical strengths
- The verifier is well-suited to a task with many valid answers. Figure 1 shows the limitations of using cell-matching to grade answers. Section 7 shows exact match is zero for every model while load-carrying varies from 33-99%, so a verifier is needed to tell models apart.
- Section 8 finds samples in v1 are close together and leave 20% of cells visible, which the authors proceed to fix. The authors also identify that scores large-grid is partly explained by a model's ability to format correctly on SPhyR-v2, which is then accounted for and reported in the evaluation metrics. Both examples show rigorous and transparent auditing of the benchmark.
Critical weaknesses
- SPhyR's entire framing is around its use as an environment for reinforcement with verifiable rewards yet no policy is ever trained against its reward. Without any training experiments the paper cannot answer if the reward is learnable or hackable as well as several other properties which are critical to assess the utility of a RL environment. This is a critical gap of the paper and would require a full set of training experiments to address. In the absence of training SPhyR is an evaluation benchmark with a sophisticated verifier.
- SPhyR is a single task environment focused on topology optimisation where there are only a few difficulty settings. As a benchmark a single environment provides narrow insight into a model's spatial-physical reasoning capability. As a training environment, it supplies one narrow skill whose utility rests on whether skill transfers to other spatial-physical tasks, which SPhyR does not test.
Questions for authors
Please can you conduct some RL training experiments with an appropriately sized model e.g. 4B-8B parameters?
Dear Reviewer 48DE,
Thank you very much for your time and effort in reviewing our work; this is highly appreciated. We would like to respond to the critical weaknesses raised by you:
If we would re-phrase the paper to be explicit about this being an evaluation environment, would that satisfy the raised weakness? Unfortunately, we might not have enough time or resources to train using this environment at this time.
Agreed, we have one environment with various tasks and difficulty levels. Can you explain why it is a requirement to expand on the tasks for this to be a valuable contribution, we would appreciate this as this will change the way we will submit this work in the future.
Could a pass@k analysis act as a substitute for this (considering potential time and resource constraints)?
Once again, thank you very much for considering our work, we will consider your feedback carefully.