- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- Proposes template-aware spreadsheet filling with slot grounding, protected regions, and local repair.
- Introduces a benchmark with instruction-only and multi-file settings.
Critical strengths
- The decomposition into grounding, validation, and local repair is clear and practical.
- The benchmark targets realistic template-preservation errors.
Critical weaknesses
- The benchmark is curated and relies partly on an LLM-based semantic judge.
- Strict form-level completion remains difficult, so robustness beyond the tested settings is unclear.
Questions for authors
- How much do results vary across runs and different spreadsheet layouts?
- Can the semantic judgments be checked with a small independent human evaluation?
3 comments
Thanks for the response. After reading through the response, I have decided to maintain my original score.
Thank you for taking the time to review our response and for your consideration. We appreciate your feedback.
Dear Reviewer c1ya,
Thank you for your thoughtful review and for recognizing the practical value of CRAFT's grounding, validation, and repair framework. We appreciate your suggestions regarding the reliability of LLM-based semantic evaluation, run-to-run variability, and generalization across spreadsheet layouts. To address these concerns, we have conducted an independent human evaluation of our semantic judge, repeated experiments to assess performance stability, and provided additional analysis across different spreadsheet structures. We hope these results offer a clearer picture of the reliability and limitations of our approach.
[W1 & Q2] Reliability of LLM-Based Semantic Evaluation
First, we would like to clarify that our LLM-based judge is used to assess semantic equivalence between predicted and ground-truth values, instead of subjective response quality. To be specific, it handles cases where semantically equivalent values have different surface forms, such as date formats and naming conventions.
To validate its reliability, we conducted an independent human evaluation. We randomly sampled 200 distinct label–slot pairs requiring LLM-based judgments across both benchmark tracks, covering predictions from CRAFT and the evaluated baselines. Each pair contained non-empty ground-truth and predicted values. Three human annotators independently classified each pair as equivalent, non-equivalent, or uncertain.
The agreement results are summarized below (see Appendix E.2 for the detailed evaluation protocol).
Among the 18 disagreements with the human majority vote, 13 were judged non-equivalent by the LLM but equivalent by humans, whereas only 5 exhibited the opposite pattern. This suggests a tendency toward more conservative semantic judgments in the sampled cases. One ambiguous instance had no human majority and was excluded from the majority-vote comparison.
Overall, these findings indicate strong agreement between the LLM judge and independent human assessments, while also revealing some ambiguity in semantic equivalence judgments. We have included the detailed evaluation protocol and results in Appendix E.2 of the revised manuscript.
[W2] Limitations and Generalizability
We agree that strict form-level completion remains challenging, as it requires correctly grounding and filling every required slot under potentially irregular spreadsheet layouts. While our additional stability and layout-wise analyses provide further evidence of CRAFT's consistency across the evaluated settings, they do not establish generalization to arbitrary unseen template structures or application domains.
We have clarified the scope of our empirical findings and will further discuss broader generalization as an important direction for future work.
[Q1] Run-to-Run Stability and Layout Variability
To assess run-to-run variability, we conducted three independent runs of CRAFT under identical experimental settings. The results are summarized below (mean ± standard deviation, %, see Table 9 in the revised Appendix E.6).
The results show relatively low run-to-run variability, particularly for Pair Accuracy on Instruction-Only. Although Multi-File exhibits greater variability in Pair Accuracy, its E2E Accuracy remains relatively consistent across runs.
We also conducted a layout-wise evaluation across five representative spreadsheet structures, as reported in Appendix E.4 (Fig. 7). CRAFT matches or outperforms the strongest baseline across all five categories, with improvements of 15 percentage points on Mixed-Slot-Layout and 12 points on Repeated-Section.
Together, these results characterize CRAFT's performance variability across repeated executions and different spreadsheet layouts. Although performance varies across settings, the observed improvements are not limited to a single layout category.