acceptodds
Under review as a conference paper at ICLR 2027

CraftBench: Benchmarking LLM-Generated Programs for Feature-Preserving Scientific Compression

Abstract

The massive volumes of data produced by scientific computing have driven widespread adoption of error-bounded lossy compression, which can achieve high compression ratios while bounding reconstruction error at every data point. However, many downstream analyses depend on derived quantities and features, such as gradients, distributions, spectra, and topological structures, that may be substantially distorted even when the pointwise error bound is satisfied. Existing feature-preserving methods are typically designed for a specific quantity or feature, or a narrow class of them, requiring substantial redesign when the target changes. Recent advances in LLM-based code generation raise the possibility of automating the adaptation of existing compressors to preserve diverse scientific features. We introduce CraftBench, a benchmark for evaluating LLM-generated adaptations of existing compressors using executable feature definitions and task-specific distance functions. CraftBench comprises 83 feature-preservation tasks spanning numerical, spatial, geometric, spectral, composite, and domain-specific analyses. Each task is evaluated under three adaptation modes: error bounding (EB), configuration selection (CS), and error correction (EC). An adaptation is considered successful only if it satisfies four criteria: (1) it reduces the distance of the target derived quantity or feature by at least 50% relative to the base reconstruction; (2) it respects the prescribed pointwise error bound; (3) it is not dominated in feature distance and compressed size by measured uniform-error-bound references, constructed by recompressing the data with progressively tighter global error bounds; and (4) its compressed output remains smaller than a fixed lossless reference. Among 2,475 LLM-generated adaptations, 282 (11.4%) satisfy all four criteria. An additional 509 satisfy the feature-distance reduction, pointwise-error, and lossless-storage requirements but are dominated by the uniform-error-bound references. These results reveal a substantial gap between generating adaptations that improve preservation of the target quantity or feature and producing adaptations that outperform the simple baseline of uniformly tightening the global error bound.

Primary area:
datasets and benchmarks
Keywords:
large language models, code generation, benchmarking, scientific data compression

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_EzrUOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CraftBench is a benchmark for evaluating whether LLMs can adapt scientific compressors to preserve specified features while satisfying error and storage constraints. It covers error-bound allocation, configuration selection, and reconstruction correction. The evaluation reveals limitations of current models in the one-shot setting and shows that many adaptations remain dominated by simply tightening the global error bound.

Critical strengths

Meaningful uniform-tightening baseline. The uniform-tightening comparison is a meaningful contribution. Improving feature preservation at a fixed error bound does not necessarily provide an advantage over simply tightening that bound uniformly. This paper makes the comparison an explicit criterion and shows that many adaptations that improve the feature are still dominated by uniform-tightening references. This baseline is useful beyond LLM evaluation and is the main reason I find the benchmark worth publishing.

General and extensible task design. Defining tasks through executable feature and distance functions is a sensible design choice that makes the benchmark adaptable to different scientific requirements. The broad feature coverage and three adaptation modes provide a useful foundation for future evaluations. The separation between task definitions, compressor interfaces, and scoring criteria also gives the benchmark good potential for reuse and extension.

Critical weaknesses

One-shot generation setting. The models generate each adaptation in one shot, without access to the data or any feedback from running the code. In practice, users would look at the result and iterate. So the low success rate may reflect the one-shot setting more than whether LLMs can produce good adaptations. It would help to clarify why the authors chose one-shot generation.

Interpretation of the overall success rate. Configuration mode measures something different from the other two modes, but the overall success rate combines all three. For SZ3, 71 of 83 tasks have no configuration that can succeed at all. For ZFP, most successes come from a single global setting that are not specific to the task. This makes the overall 11.4% hard to interpret. I’d suggest reporting configuration mode separately, or computing the SZ3 configuration rate only on tasks where a successful configuration exists.

Questions for authors
  1. How much work would it take to add another compressor (e.g., MGARD or SPERR) or a new feature? It would be useful to know how easily others can extend the benchmark.

  2. Could CraftBench also be used in a setting where the model can see how its adaptation performs and then revise it? Have you tried this?

2 comments

Thank you for the comments. In summary, (1) we evaluate one-shot generation by design, and a small pilot shows that evaluator feedback can further improve success (17 of 42 failed conditions repaired, vs. 13 with resampling); (2) the low success rate of configuration mode is itself a finding, and we now also report the rate for error-map and correction modes alone (13.2%); (3) we clarify what is needed to add a new compressor or feature; and (4) we summarize the strategies that successful adaptations share. Below, we respond to each point in the order raised in the review.

  1. Clarified One-Shot Evaluation and Evaluated Feedback (Pilot Study). We evaluate single-response generation by design, so that results reflect what a model can derive from the task specification alone, without confounding from search budgets or the number of feedback rounds. Although the model does not see the data values during generation, the generated error-map and correction functions take the original field as input when they run, so these adaptations can still adapt to the data. Multi-round settings are a natural extension, since CraftBench’s executable evaluator can directly provide feedback to the model. In a small pilot on 42 originally failed conditions, evaluator feedback repairs 17 within two revisions, compared with 13 for resampling without feedback (Appendix F.16). We leave a full evaluation to future work (Section 6).

  2. Clarified Configuration-Mode Interpretation. We include configuration mode to measure how far feature preservation can go by selecting existing compressor settings alone, without changing error allocation or reconstructed values. Its low success rate is itself a finding: for SZ3, our exhaustive enumeration (Appendix F.6) shows that only 10 of 83 tasks admit any successful configuration, and on these tasks 16 of 50 generated configurations succeed (32.0%). For ZFP, Appendix F.7 shows that a single fixed configuration also achieves 31 of Kimi-K2.5's 32 configuration successes. Because combining configuration mode with the other modes makes the overall rate harder to interpret, Section 5.3 now also reports the overall rate for error-map and correction modes alone: 218 of 1,650 (13.2%).

  3. Response on Extending CraftBench to New Features and Compressors. Adding a feature only requires writing its feature and distance functions; prompts, reference sweeps, and evaluation are generated automatically from these. Adding a compressor requires a wrapper for compression and decompression, support for per-point or per-block error bounds (for error-map mode), a configuration schema (for configuration mode), and the uniform-reference sweep. The rest of the pipeline is unchanged. We will document this in the released code.

  4. Response on Common Strategies Among Successful Adaptations. Yes. Successful corrections usually find the points where the feature is actually affected, for example, points whose threshold classification differs between the original and reconstructed fields, and fix only the most important ones, as in the example in Appendix F.1. Successful error maps similarly tighten bounds near feature-relevant values, such as thresholds, or in high-gradient regions (45 of 97 successful error maps). In contrast, failed corrections modify more points and store more correction data (medians of 512 vs. 189 points and 2,057 vs. 663 bytes; Appendix F.4). These patterns resemble the correction-based and error-map-based designs in prior feature-preserving work, which suggests the benchmark may help identify promising strategies for new features.

Thank you for the detailed responses. My main concerns have been addressed, and I find CraftBench to be a meaningful benchmark for evaluating feature-preserving compressor adaptations. I have raised my rating from 3 to 4.

Reviewer_S4izOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper constructed a benchmark in the AI applications to lossy compression of scientific data.

Lossy compressions may acquire a more compact representation of the orginal numerical data by introducing a point-wise controlled error. Major lossy compressors like SZ3 or ZFP are designed to use certain connectivity or regulairty properties of the data in "recording fewer important values" to compress. But the compressors usually aim at control the absolute error in compression, and certain controlled errors in data may affect the data features such as gradient, spectrum, topology, etc. Practically, the compressor must combine three strategies: controlling the error bound, choosing configuration of the compressor, and saving a correction term. In this benchmark, LLMs are employed to generate these three terms.

The benchmark contains 83 tasks on controlling histogram, gradient, spectrum, Euler characteristic, etc. Four criteria are constructed to define success of each task, a compression that satisfies all four criteria is considered successful. They are:(C1) reduction in target feature distance, (C2)compliance with the pointwise error bound, (C3) non-domination by uniform-error-bound references, and (C4) compressed size smaller than the lossless reference.

Five LLMs are evaluated on the benchmark. Overall, 282 of 2,475 adaptations (11.4%) succeed, with model-level rates ranging from 4.4% to 17.2%.

Critical strengths
  1. The topics (feature-preserving / topology-preserving compression) is new in AI (though not new in HPC / compression field), and in terms of this benchmark, the tasks that this benchmark define are critical for current LLMs.

  2. The 4 criteria are well designed. The creteria altogether form an effective condition on successfulness, and effectively ruled out certain fake positives.

  3. The benchmark success rate baseline is fair. For example, the paper checked achievabiliity under the compressor configuration spaces. Some low successful-rate results may just caused by the large size of unachievable space.

Critical weaknesses
  1. Sometimes for scientific data compressions, some research could consider to define more flexible improvements, such as using adaptive mesh refinement, etc., together with certain lossy compressor. These are also capable for modern LLMs, could the benchmark be enriched to cover these kind of possible solutions?

  2. In evaluation, could you let the LLM generate solutions more than once? In other benchmarks such as GPQA or simply Math500, usually pass@k are used, maybe you can also add the results of pass@k for k>1k>1.

  3. In evaluation, LLMs cannot read the data values directly, but only writing certain codes for analysis. Could it be relaxed to some more flexible restrictions? I think this might also help the LLMs to work better.

Questions for authors
  1. The raw data (the base scientific computation data to be compressed) seem to be restricted on fluid dynamics and fields in fluid. I am wondering whether such data are representative enough for the whole research field?
1 comment

Thank you for the positive assessment. The new results support the paper’s main finding and show how the benchmark can be extended: the success rate stays at 11.0%–13.7% across additional fields and error bounds, and a small pilot shows that repeated sampling and evaluator feedback can repair failed adaptations. We respond to each point below in the order raised in the review.

  1. Clarification on Broader Adaptation Interfaces (AMR). Yes. The success criteria depend only on the final reconstruction data and the total stored bytes, so adaptations such as AMR combined with a lossy compressor can be evaluated with the same protocol. Supporting them requires a new adaptation interface in which the generated program outputs a refined or resampled representation, with any mesh metadata counted in the stored bytes. We kept the current study to three interfaces that adapt a fixed compressor without changing its data representation, so that results are comparable across tasks. Appendix F.10 (Table 13) lists AMR and other similar methods as extensions outside the current scope.

  2. Newly Evaluated Independent Resampling (Pilot Study); Full Pass@k Remains Future Work. Section 5.2 shows that 34.9% of settings are solved by at least one of the five models; this is cross-model coverage from one generation per model, not pass@k. As a direct test of repeated sampling, we ran a small pilot on 42 conditions where the original single generation of GPT-5.3-Codex failed: two new independent samples solve 13 of them (8 with the first sample), indicating that single-generation rates underestimate what repeated sampling can achieve (Appendix F.16). We leave a full pass@k evaluation to future work (Section 6).

  3. Clarification on Data Access and Pilot Evaluation of Feedback. Yes, this restriction can be relaxed, and it may improve performance. Since prompts are generated from the task specification, CraftBench can include data summaries (e.g., value ranges, histograms, or a downsampled field) in the prompt, or return evaluation results to the model for revision. In the current study, we keep generation data-free to test whether models can turn a feature definition into a strategy without fitting to a single field. The generated functions still take the original field as input when they run, so they can compute data-dependent quantities, such as thresholds from the field’s mean and standard deviation. In a small pilot, returning evaluation results to the model repairs 17 of 42 originally failed conditions within two revisions, compared with 13 for resampling without feedback (Appendix F.16). Section 6 notes a full evaluation of these relaxed settings as future work.

  4. Expanded Evaluation Across NCEP Time Slices and Error Bounds. The evaluation fields come from three domains: climate (NCEP/NCAR air temperature), cosmology (Nyx density), and turbulence (TACC enstrophy). Appendix F.9 additionally evaluates on a much larger GFS temperature field. We also re-evaluated all saved adaptations, without new model calls, on four additional days of the NCEP field and at error bounds of e₀/10 and 10e₀. Success rates range from 11.0% to 13.7% across these settings, compared with 11.4% (95% CI 9.4%–13.5%) in the paper, and the model ranking is the same in every setting (Appendix F.13, Table 16).

Reviewer_LLCoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CraftBench evaluates LLM-generated adaptations of scientific compressors across 83 feature-preservation tasks, two backends, and three adaptation modes. Success requires improved feature fidelity, compliance with pointwise error bounds, non-domination by measured uniform-tightening references, and storage below a lossless baseline. Five models produce 2,475 adaptations, of which 11.4% succeed. Additional analyses examine failures, configuration feasibility, and transfer to a larger temperature field.

Critical strengths
  • The benchmark goes beyond executable correctness by jointly evaluating feature preservation and complete storage cost, including adaptation metadata. The uniform-tightening comparison provides a meaningful check against improvements achievable through simpler settings.

  • The feasibility and fixed-configuration analyses provide useful negative findings and candidly expose limitations of the headline success rate.

Critical weaknesses
  • Scientific relevance is insufficiently validated. Despite 83 feature tasks, 74 use the same 73×14473\times144 temperature field. The larger-field experiment helps, but the evaluation remains narrow. More importantly, the paper does not show when meeting its feature-distance thresholds preserves a consequential downstream scientific result. Diverse executable descriptors do not, by themselves, establish diverse scientific utility.

  • Success does not establish useful adaptation. Only 10/83 SZ3 configuration tasks are feasible, while a post hoc fixed ZFP configuration succeeds on more tasks than Kimi’s adaptive selections (34/82 versus 32/82). Thus, aggregate success mixes interface limitations with adaptation ability. Compression/decompression throughput and per-field adaptation overhead are also unreported. Without stronger simple baselines and runtime measurements, it remains unclear whether successful adaptations offer a practical advantage.

Questions for authors
  1. Can you demonstrate a representative downstream scientific analysis whose conclusion is preserved by meeting your feature criteria?

  2. What advantage remains over simple heuristics or configuration search after accounting for task feasibility?

  3. What are the compression and decompression throughputs, including per-field adaptation overhead, relative to the base compressor and uniform-tightening references?

  4. How are boundary-touching background components handled in the Euler-characteristic implementation (p. 24)? As written, a solid stripe spanning opposite boundaries gives 0, although it has one component and no holes.

  5. Table 10’s counts sum to all 825 EC attempts, but its caption refers to valid correction instances. Does this count include compilation and runtime failures? What sample sizes underlie the reported medians?

  6. Section 5.5 reports that only 10 of 282 successes become dominated under the extended reference sweep, whereas Appendix F.8 extends the sweep for only 31 candidates. What justifies excluding the remaining candidates, and what conclusion is supported for the full set?

1 comment

Thank you for the detailed review and the careful reading of the appendix. The additional experiments confirm that the main result is robust: it holds across four additional temporal slices of the NCEP field and two additional error bounds, under a sweep extended to e₀/4096, and after fixing the Euler characteristic implementation, which leaves the overall result unchanged. We respond to each point below in the order raised in the review.

  1. Expanded Evaluation Across NCEP Time Slices and Error Bounds. We re-evaluated all saved adaptations, without new model calls, on four additional days of the NCEP field (January, April, July, and October) and at error bounds of e₀/10 and 10e₀. Success rates range from 11.0% to 13.7% across these settings, compared with 11.4% (95% CI 9.4%–13.5%) in the paper, and the model ranking is the same in every setting. We add these results in Appendix F.13 (Table 16), alongside the larger GFS field in Appendix F.9.

  2. Added a Downstream Scientific Example. For the high-temperature exceedance task with SZ3, we compared three fields on a simple downstream quantity, the number of grid points above 303.15 K and the centroid of the largest above-threshold region: the original data, the reconstruction from SZ3 at e₀, and the reconstruction from a successful correction generated by Claude-Sonnet-5. The base reconstruction increases the count from 78 to 87 and shifts the centroid, while the adapted reconstruction (a sparse correction) restores both exactly. We add this example in Appendix F.15 (Figure 6).

  3. Added Task-Agnostic Baselines and Configuration Comparisons. We added task-agnostic heuristics that apply the same rule to every task. The error-map heuristic sets each point’s bound inversely proportional to its gradient magnitude, clipped to [e₀/64, e₀]; the correction heuristic stores the exact values of the 1% of points with the largest reconstruction error. All heuristics are reported in Appendix F.11 (Table 14).

    a. In correction mode, LLM-generated adaptations achieve a 14.7% average success rate, compared with 3.6% for the best individual task-agnostic correction heuristic.

    b. In error-map mode, the gradient-based heuristic beats all models on ZFP (40.2% vs. 24.4%), but no heuristic succeeds on SZ3 (best model 7.2%).

    c. In configuration mode, the advantage is limited, as shown by the fixed ZFP configuration (Appendix F.7) and the 10 of 83 feasible SZ3 tasks. Section 5.3 now also reports the overall success rate without configuration mode, which is 13.2%.

  4. Added Runtime and Throughput Measurements. Appendix F.12 (Table 15) now reports in-process compression and decompression times and throughput (original 42,048-byte field divided by time) for the base compressor, uniform-tightening references, and adapted methods on 18 representative successful cases measured on the same node. For each case, the uniform comparator is the cheapest uniform setting achieving equal or better feature distance, or the tightest sweep point when no such setting exists. Eleven of the 18 cases have a matching lossy uniform reference, while the remaining seven use the tightest sweep point as a fallback. Uniform tightening takes 1.2–1.6× the base encoding time and 1.3–3.0× the base decoding time. Adapted configurations run close to base-compressor speed (e.g., SZ3: 0.448 / 0.194 ms vs. base 0.459 / 0.186 ms). Error-map and correction adaptations introduce additional encoding overhead, while applying corrections during decoding adds approximately 0.01 ms. All measurements remain in the millisecond range for these 73 × 144 fields.

  5. Corrected Euler Characteristic Implementation. We fixed the implementation so that only background components that do not touch the domain boundary are counted as holes, and so that the complement uses the dual connectivity. We updated the definition in Appendix B.1.4. After re-evaluating all 30 candidates and 50 uniform references for this task, all seven original successes remain successful, none is gained or lost, and the overall result is unchanged at 282/2,475. We thank the reviewer for catching this.

  6. Corrected Table 10 Caption and Denominators. We corrected the caption and added the number of instances behind each median (Appendix F.4, Table 10). The table counts all 825 correction attempts: 121 successful, 639 valid but unsuccessful, and 65 invalid. The median of 512 modified points is computed over the 639 instances with measurements.

  7. Extended Uniform-Reference Sweep. We extended the uniform sweep to e₀/4096 for all 282 successful candidates, not only the 31 analyzed previously. 272 remain non-dominated, consistent with the 10 reported in the paper. We updated Section 5.5 and Appendix F.8 accordingly.

Reviewer_FYXUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses a known problem in scientific data compression: compressing the raw data accurately, with every value within a pointwise error bound, does not guarantee that key features derived from the data, such as gradients, topological structures or spectra, are preserved. To study whether LLMs can help, it introduces CraftBench, an execution-based evaluation of LLM-generated adaptations of two error-bounded lossy compressors, SZ3 and ZFP. Each of 83 tasks pairs a fixed scientific field (mostly one 73×144 climate slice, plus two 128³ fields) with an executable feature function, a task-specific distance and a pointwise error bound e0. In one single-shot generation, without seeing the data values, an LLM writes an adaptation in one of three modes: a per-point error-bound map before compression (EB), a choice of compressor settings (CS), or stored corrections applied after decompression (EC). An adaptation succeeds only if it meets four binary criteria: it halves the feature distance relative to plain compression at e0, keeps the pointwise bound, is not dominated in (feature distance, size) by a sweep of uniformly tighter error bounds, and stays smaller than lossless compression. The contributions are the task suite, the compression-aware evaluation protocol (notably the uniform-tightening baseline), and an evaluation of five LLMs over 2,475 instances. Validation is empirical: 11.4% of adaptations succeed (4.4–17.2% by model), and 509 more improve the feature but are matched by uniform tightening. The authors add an exhaustive search of the SZ3 configuration space (only 10 of 83 tasks are solvable in that mode), a fixed-configuration check for ZFP, and a breakdown of failures by criterion.

Critical strengths

While I am not an expert in this area, the use of LLMs for scientific compression appears to be a frontier topic, and the paper is a timely, well-grounded attempt to study it. With respect to specific technical strengths I would highlight:

  1. The uniform-tightening baseline. The evaluation asks whether an adaptation beats simply tightening the global error bound, not just whether it improves the feature. This is the paper's most valuable idea: it separates real feature-specific gains from spending more bits, and it shows that most apparent improvements (509 of them) are matched by that trivial strategy. It's consequential because it's a reusable evaluation principle for feature-preserving compression, valuable beyond this benchmark.
  2. Executable, problem-relevant task definitions with honest self-analysis. Each task specifies its feature and distance as code over real scientific fields, which makes the evaluation concrete and reproducible. The authors also test their own results (the exhaustive SZ3 configuration search, the fixed-configuration check that explains Kimi's ZFP results, the extended reference sweep) and state their limitations openly. That raises my confidence in the measurements themselves, even where I question their interpretation.
Critical weaknesses
  1. Framing and scope: an unclear object of study and no grounding. Restricting the study to LLM-generated adaptations is a reasonable choice, but the paper should state it explicitly: what is being evaluated is single-shot LLM generations under three fixed interfaces, and why LLMs are a promising fit for this problem. Instead, "adaptation" is never defined, so the three modes (EB, CS, EC) appear without justification, and "CraftBench" is used both for the benchmark and for the methods it evaluates (Figure 2 places the LLM inside the benchmark). The paper would benefit from a clear functional formalization of the class of algorithms evaluated. In our reading, every evaluated method has the form ŷ = f(x̂): the feature function f is fixed and only the reconstruction x̂ changes, through a per-point error-bound map (EB), a choice of compressor settings (CS), or stored corrections (EC), always within the pointwise bound. The paper should make this explicit and say which types of adaptation are and are not evaluated: for example, methods that modify the compressor's code, that loosen the bound where the feature is insensitive, that store features directly, or that iterate with evaluator feedback. Even with LLMs as the subject, the results also need reference points from methods of the same form: a simple heuristic, a human-written adaptation, or existing feature-preserving compressors (TopoSZ, QPET, edit-based correction, MGARD's quantity-of-interest bounds) on matching tasks. None are included. Without them, the headline 11.4% has no reference point: we cannot tell whether it is low, high, or simply what the task allows (in SZ3 configuration mode, 71 of 83 tasks admit no successful configuration). This matters most to my assessment because it determines what the paper's main result means.
Questions for authors

I have no clarification questions that would change my assessment. My comments are suggestions for future work.

In its present form I do not think the paper is ready for publication. But the authors have done the hard work: the task suite, the executable feature definitions, the uniform-tightening baseline, and a large set of experiments. Clarifying the paper's intent, polishing the presentation, and running a proper statistical analysis that brings out the richness of the data could make it a good publishable paper. Specific suggestions:

  1. State explicitly what is evaluated, with a functional formalization of the adaptations considered and those excluded.
  2. Add reference points: a simple heuristic, a human-written adaptation, and existing feature-preserving methods on matching tasks.
  3. Run an analysis of variance, or a similar approach suited to binary outcomes such as mixed-effects logistic regression, that clarifies whether the factors (model, adaptation mode, compressor) have uniform effects or significant interactions. Do the same for tasks: whether some model and mode combinations work on specific tasks or uniformly across them. This requires multiple generations per instance and confidence intervals on all reported rates.
  4. Report the feature-distance and size tradeoff continuously (for example, distance to the uniform-tightening frontier) rather than only as binary success.
  5. Explain the regime that motivates the pointwise bound, and consider results with and without it.
  6. Broaden the data beyond a single 73×144 slice and a single error bound.
  7. Consider an iterative mode with evaluator feedback, the natural next step for LLM-based adaptation.
2 comments
AuthorsOpenReview ↗

Response to FYXU (Part 1/2): Formalization and Baselines

Thank you for the detailed review and for recognizing the uniform-tightening baseline as a reusable evaluation principle. The new reference points show what the 11.4% means: where a task requires reasoning about the target feature, as in correction mode, LLMs clearly outperform task-agnostic rules, while in other cases they miss simple strategies that work. We first address the framing and reference points, then each of the seven suggestions.

  1. Enhanced Benchmark Formalization. We adopt the reviewer’s formalization with one clarification. Every evaluated adaptation changes only the reconstruction: the feature function f is fixed and evaluated on the reconstruction x̂, which changes only through an error-bound map, a compressor configuration, or stored corrections. Compliance with the prescribed pointwise error bound ‖x − x̂‖∞ ≤ e₀ is enforced as a success criterion rather than assumed, so adaptations that violate it are counted as failures. Section 4.2 now states this explicitly, Appendix F.10 (Table 13) lists the adaptation types outside the main evaluation (modifying compressor code, loosening the bound, storing features directly, adaptive mesh refinement, and multi-round generation with feedback, for which Appendix F.16 reports a small pilot), and the Figure 2 caption clarifies that the LLM is the system under evaluation rather than part of the benchmark.

  2. Added Task-Agnostic Heuristic Baselines. We added task-agnostic heuristics that apply the same rule to every task: the strongest error-map heuristic sets each point’s bound inversely proportional to its gradient magnitude, clipped to [e₀/64, e₀], and the correction heuristic stores the exact values of the 1% of points with the largest reconstruction error.

Mode / backendTask-agnostic baselineLLMs, averageBest LLM
Error map / SZ30.0% (gradient-inverse heuristic)3.6%7.2%
Error map / ZFP40.2% (gradient-inverse heuristic)20.0%24.4%
Configuration / SZ38.4% (best single configuration)3.9%8.4%
Configuration / ZFP41.5% (fixed expert configuration)11.7%39.0%
Correction / SZ31.2% (top-1% residual correction)13.5%32.5%
Correction / ZFP6.1% (top-1% residual correction)15.9%29.3%

These reference points show what the benchmark measures. In correction mode, LLM-generated adaptations achieve a 14.7% average success rate, compared with 3.6% for the best individual task-agnostic heuristic (4.2% for the oracle union of tested correction heuristics). Where a simple rule already works, as for gradient-based error maps on ZFP, current models often fail to find it. The 11.4% therefore reflects real task difficulty together with specific, identifiable gaps in current models. Section 5.3 summarizes these results, and Appendix F.11 (Table 14) reports all heuristics.

Existing feature-preserving methods target specific feature classes: TopoSZ focuses on contour-tree topology and extrema, while QPET preserves differentiable quantities of interest. Although they overlap with some CraftBench tasks, their preservation objectives and guarantees do not directly correspond to CraftBench’s executable feature and distance definitions. We therefore use task-agnostic heuristics as reference points evaluated under the same criteria across all eligible tasks, and discuss the relationship to feature-specific methods in Appendix F.10.

AuthorsOpenReview ↗

Responses to the Seven Suggestions

Response to each suggestion.

  1. Enhanced Benchmark Formalization. Added in Section 4.2 and Appendix F.10, as above.

  2. Added Baselines and Discussed Existing Feature-Preserving Methods. Heuristic baselines and fixed configurations added, as above. Appendix F.10 also discusses how existing feature-preserving methods, such as edit-based correction for contour trees and extremum graphs and QPET, relate to CraftBench tasks.

  3. Added Statistical Analysis. Section 5.2 now reports a 95% bootstrap confidence interval over tasks (11.4%, CI 9.4%–13.5%), and Appendix F.14 adds a mixed-effects logistic regression with task as a random effect. Model, mode, and backend all have significant main effects (p < 0.001), and the mode × backend interaction is significant (p < 10⁻⁶), meaning the best adaptation mode depends on the compressor. The model × mode interaction cannot be estimated reliably because one model never succeeds in configuration mode. Appendix F.16 adds a small pilot with repeated generations, and Section 6 notes a full multi-generation evaluation as future work.

  4. Expanded Storage–Fidelity Tradeoff Analysis. Section 5.2 now reports that successful adaptations use a median of 27.4% fewer bytes than the cheapest reference on the uniform rate–distortion curve, including its lossless endpoint, that achieves equal or better feature distance (n = 282). Restricting to the 206 cases where a lossy uniform setting reaches that distance, the median saving is 18.0% (IQR 10.9%–33.4%); for the remaining 76, no lossy uniform setting in the sweep matches the adaptation's feature distance (Section 5.2).

  5. Clarification on Pointwise Error Bounds and Scoring Sensitivity. Error-bounded compressors are widely used because many scientific workflows require a guaranteed per-point error, so we keep this constraint. Dropping it changes the number of successes only from 282 to 283 (Appendix F.10).

  6. Expanded Evaluation Across NCEP Time Slices and Error Bounds. We re-evaluated all saved adaptations, without new model calls, on four additional days of the NCEP field and at error bounds of e₀/10 and 10e₀. Success rates range from 11.0% to 13.7%, compared with 11.4% in the paper, and the model ranking is the same in every setting (Appendix F.13, Table 16). Appendix F.9 additionally evaluates a larger GFS field.

  7. Newly Evaluated Repeated Sampling and Feedback (Pilot Study). We ran a small pilot on 42 conditions where the original single generation of GPT-5.3-Codex failed (Appendix F.16). With evaluator feedback, the model repairs 15 of them with one revision and 17 with two, compared with 8 and 13 when drawing the same number of new samples without feedback. The feedback includes information withheld from the one-shot prompt, such as the lossless reference size and whether the candidate is dominated, so part of this gain reflects access to that information rather than revision alone. A full evaluation is left to future work (Section 6).

We hope these additions clarify the scope of the study and the meaning of the main result, and we are happy to provide further details.

Reviewer_pLguOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces CraftBench, a benchmark for evaluating how well LLMs can adapt existing scientific data compressors to preserve features specified through executable code. It includes 83 feature-preservation tasks and evaluates three types of adaptation: error bounding, configuration selection, and error correction. The evaluation considers feature error, pointwise error constraints, and storage cost. The authors also provide a detailed analysis of the results.

Critical strengths
  1. The paper explores a new LLM-based approach to adapting scientific data compressors for feature preservation.
  2. The benchmark and evaluation criteria provide a useful basis for further research on this problem.
  3. The paper provides a thorough analysis of both successful and unsuccessful adaptations.
Critical weaknesses
  1. The contribution mainly lies in constructing the benchmark. The study provides limited new insight into the principles or methods of feature-preserving compression.
  2. The experiments do not establish a clear benefit from task-specific configuration selection: Kimi succeeds on 32/82 ZFP instances, compared with 34/82 for the fixed configuration.
  3. The paper does not report the additional encoding and decoding costs, leaving the practical trade-off between feature preservation, storage, and computation unclear.
Questions for authors
  1. What common strategies appear in the successful adaptations? Please use one or two existing examples to explain how these strategies depend on the target feature.
  2. Can you report the encoding and decoding times for a few representative adaptations, compared with the base compressor?
1 comment

Thank you for the review and the concrete questions. Beyond the benchmark, the new results clarify where generated adaptations help and what they cost: task-specific corrections clearly outperform simple rules, successful adaptations follow recognizable strategies, and the added encoding and decoding time is reported in Appendix F.12. We respond to each point below in the order raised in the review.

  1. Response on Insights into Feature-Preserving Compression. Beyond the benchmark itself, the results offer three insights into feature-preserving compression.

    a. First, improving a feature at a fixed error bound does not by itself show that a method is useful: 509 adaptations improve the target feature, yet simply tightening the global error bound achieves the same or better preservation at no larger size, so feature-preserving methods, including hand-designed ones, should also be compared against uniform tightening.

    b. Second, our new heuristic baselines (Appendix F.11) show that allocating tighter error bounds to high-gradient regions is effective on ZFP: this single rule succeeds on 40.2% of ZFP error-map instances, although it does not succeed on SZ3. This indicates that, for ZFP, many features are most sensitive to errors where the field changes rapidly.

    c. Third, successful adaptations locate where the target feature is affected and apply tighter error bounds or corrections there, whereas failed corrections modify more points and store more correction data (medians of 512 vs. 189 points and 2,057 vs. 663 bytes).

  2. Clarification on Configuration Selection and Fixed-Configuration Comparisons. This comparison is one reason we report modes separately: configuration mode mainly tests the limits of existing settings, while task-specific reasoning pays off in correction mode, where LLM-generated corrections achieve a 14.7% average success rate, compared with 3.6% for the best individual task-agnostic correction heuristic (Table 14). We reported the Kimi comparison ourselves (Section 5.3, Appendix F.7) to avoid overstating configuration results, and Section 5.3 now also reports the overall rate without configuration mode (13.2%).

  3. Response on Common Strategies in Successful Adaptations. Successful adaptations locate where the target feature is affected and apply tighter error bounds or corrections there. Two examples show how this depends on the target feature (Appendix F.1):

    a. Topological merge level (SZ3, error map): the adaptation computes persistence pairs from the original field and tightens bounds only around the most persistent peaks and their merge saddles. It uses 67.5% fewer bytes than the matching uniform reference.

    b. Skewness and kurtosis (SZ3, correction): since the target is the distribution’s tails, the adaptation ranks points by reconstruction error weighted by distance from the mean and corrects tail values first. It uses 28.7% fewer bytes than the matching uniform reference.

    Across all successes, nearly half of the successful error maps (45 of 97) allocate tighter bounds based on gradients or derivatives, while successful corrections mostly store sparse residuals or fix points whose threshold classification changed.

  4. Added Encoding/Decoding Time and Throughput Measurements. Appendix F.12 (Table 15) now reports in-process encoding and decoding times and throughput for 18 representative NCEP cases, comparing the base compressor, uniform-tightening references, and generated adaptations. Adapted encoding takes 0.211–2.184 ms, and decoding takes 0.153–0.634 ms, compared with 0.171–0.499 ms and 0.119–0.187 ms for the base compressor. Uniform tightening takes approximately 1.2–1.6× the base encoding time and 1.3–3.0× the base decoding time. Applying corrections during decoding adds approximately 0.01 ms. These measurements include adaptation overhead and exclude one-time compilation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their constructive feedback. We have revised the manuscript with the following clarifications, corrections, experiments, and analyses.

  1. Clarified Success-Rate Interpretation and Evaluated Robustness. Of 2,475 generated adaptations, 282 (11.4%) satisfy all four success criteria, while another 509 meet the feature-improvement, pointwise-error, and storage requirements but are dominated by uniform tightening. Re-evaluating saved adaptations on four additional temporal slices of the NCEP field and under different error bounds yields success rates of 11.0%–13.7% (Appendix F.13). Extending the uniform-reference sweep to e₀/4096 leaves 272 of 282 successes non-dominated (Appendix F.8).

  2. Added Task-Agnostic Baselines and Clarified Mode Differences. Against task-agnostic heuristics (Appendix F.11), LLM-generated corrections achieve a 14.7% success rate, compared with 3.6% for the best individual heuristic. Conversely, on ZFP error maps, a simple gradient-based heuristic outperforms all evaluated models. Excluding configuration mode, the overall success rate is 13.2%.

  3. Expanded Storage–Fidelity and Runtime Evaluation. Successful adaptations use a median of 27.4% fewer bytes than the cheapest uniform reference achieving equal or better feature distance, including the lossless endpoint (n = 282). Among the 206 cases with matching lossy uniform references, the median saving is 18.0%. Table 15 reports encoding and decoding times and throughput against base compressors and uniform-tightening references. Appendix F.15 presents a downstream example in which an adapted reconstruction restores the original above-threshold count and region centroid.

  4. Newly Evaluated Repeated Sampling and Feedback (Pilot Study). Appendix F.16 reports a pilot on 42 originally failed conditions: two additional independent samples solve 13, while evaluator feedback repairs 17 within two revisions. A full multi-generation evaluation remains future work.

We address the reviewers' specific questions in our individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.