- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
CraftBench is a benchmark for evaluating whether LLMs can adapt scientific compressors to preserve specified features while satisfying error and storage constraints. It covers error-bound allocation, configuration selection, and reconstruction correction. The evaluation reveals limitations of current models in the one-shot setting and shows that many adaptations remain dominated by simply tightening the global error bound.
Critical strengths
Meaningful uniform-tightening baseline. The uniform-tightening comparison is a meaningful contribution. Improving feature preservation at a fixed error bound does not necessarily provide an advantage over simply tightening that bound uniformly. This paper makes the comparison an explicit criterion and shows that many adaptations that improve the feature are still dominated by uniform-tightening references. This baseline is useful beyond LLM evaluation and is the main reason I find the benchmark worth publishing.
General and extensible task design. Defining tasks through executable feature and distance functions is a sensible design choice that makes the benchmark adaptable to different scientific requirements. The broad feature coverage and three adaptation modes provide a useful foundation for future evaluations. The separation between task definitions, compressor interfaces, and scoring criteria also gives the benchmark good potential for reuse and extension.
Critical weaknesses
One-shot generation setting. The models generate each adaptation in one shot, without access to the data or any feedback from running the code. In practice, users would look at the result and iterate. So the low success rate may reflect the one-shot setting more than whether LLMs can produce good adaptations. It would help to clarify why the authors chose one-shot generation.
Interpretation of the overall success rate. Configuration mode measures something different from the other two modes, but the overall success rate combines all three. For SZ3, 71 of 83 tasks have no configuration that can succeed at all. For ZFP, most successes come from a single global setting that are not specific to the task. This makes the overall 11.4% hard to interpret. I’d suggest reporting configuration mode separately, or computing the SZ3 configuration rate only on tasks where a successful configuration exists.
Questions for authors
-
How much work would it take to add another compressor (e.g., MGARD or SPERR) or a new feature? It would be useful to know how easily others can extend the benchmark.
-
Could CraftBench also be used in a setting where the model can see how its adaptation performs and then revise it? Have you tried this?
2 comments
Thank you for the detailed responses. My main concerns have been addressed, and I find CraftBench to be a meaningful benchmark for evaluating feature-preserving compressor adaptations. I have raised my rating from 3 to 4.
Thank you for the comments. In summary, (1) we evaluate one-shot generation by design, and a small pilot shows that evaluator feedback can further improve success (17 of 42 failed conditions repaired, vs. 13 with resampling); (2) the low success rate of configuration mode is itself a finding, and we now also report the rate for error-map and correction modes alone (13.2%); (3) we clarify what is needed to add a new compressor or feature; and (4) we summarize the strategies that successful adaptations share. Below, we respond to each point in the order raised in the review.
Clarified One-Shot Evaluation and Evaluated Feedback (Pilot Study). We evaluate single-response generation by design, so that results reflect what a model can derive from the task specification alone, without confounding from search budgets or the number of feedback rounds. Although the model does not see the data values during generation, the generated error-map and correction functions take the original field as input when they run, so these adaptations can still adapt to the data. Multi-round settings are a natural extension, since CraftBench’s executable evaluator can directly provide feedback to the model. In a small pilot on 42 originally failed conditions, evaluator feedback repairs 17 within two revisions, compared with 13 for resampling without feedback (Appendix F.16). We leave a full evaluation to future work (Section 6).
Clarified Configuration-Mode Interpretation. We include configuration mode to measure how far feature preservation can go by selecting existing compressor settings alone, without changing error allocation or reconstructed values. Its low success rate is itself a finding: for SZ3, our exhaustive enumeration (Appendix F.6) shows that only 10 of 83 tasks admit any successful configuration, and on these tasks 16 of 50 generated configurations succeed (32.0%). For ZFP, Appendix F.7 shows that a single fixed configuration also achieves 31 of Kimi-K2.5's 32 configuration successes. Because combining configuration mode with the other modes makes the overall rate harder to interpret, Section 5.3 now also reports the overall rate for error-map and correction modes alone: 218 of 1,650 (13.2%).
Response on Extending CraftBench to New Features and Compressors. Adding a feature only requires writing its feature and distance functions; prompts, reference sweeps, and evaluation are generated automatically from these. Adding a compressor requires a wrapper for compression and decompression, support for per-point or per-block error bounds (for error-map mode), a configuration schema (for configuration mode), and the uniform-reference sweep. The rest of the pipeline is unchanged. We will document this in the released code.
Response on Common Strategies Among Successful Adaptations. Yes. Successful corrections usually find the points where the feature is actually affected, for example, points whose threshold classification differs between the original and reconstructed fields, and fix only the most important ones, as in the example in Appendix F.1. Successful error maps similarly tighten bounds near feature-relevant values, such as thresholds, or in high-gradient regions (45 of 97 successful error maps). In contrast, failed corrections modify more points and store more correction data (medians of 512 vs. 189 points and 2,057 vs. 663 bytes; Appendix F.4). These patterns resemble the correction-based and error-map-based designs in prior feature-preserving work, which suggests the benchmark may help identify promising strategies for new features.