acceptodds
Under review as a conference paper at ICLR 2027

CRAFT: An Agentic Spreadsheet Form Filling System with Template Awareness

Abstract

Spreadsheet form filling requires agents to consolidate external evidence, ground values to precise cells, and preserve irregular template structure. Errors in early edits can overwrite labels or misalign fields, undermining later decisions. We propose CRAFT, a template-aware agent framework that connects reflective validation to constrained local repair. Instead of treating reflection as a free-form request to regenerate the workbook, CRAFT grounds detected errors to spreadsheet regions, restores corrupted template state when necessary, and re-grounds plausible writable slots before subsequent edits. A Rectangle-Aware Slot Grounder (RASG) proposes writable cells, while label–slot hints and protected regions constrain subsequent edits. We introduce FormFillBench, with 327 forms across Instruction-Only and Multi-File tracks. Compared with the strongest baselines, CRAFT improves pair accuracy by 8.51 and 23.38 percentage points on these tracks, respectively. Component-removal experiments support structural adjudication and slot re-grounding within the pipeline, and the framework retains its relative advantage among the methods evaluated with a second backbone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Spreadsheet Form Filling, Multimodal Document Understanding, LLM Agents, Structured Reasoning, Slot Grounding, Error Detection and Repair

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_c1yaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  • Proposes template-aware spreadsheet filling with slot grounding, protected regions, and local repair.
  • Introduces a benchmark with instruction-only and multi-file settings.
Critical strengths
  • The decomposition into grounding, validation, and local repair is clear and practical.
  • The benchmark targets realistic template-preservation errors.
Critical weaknesses
  • The benchmark is curated and relies partly on an LLM-based semantic judge.
  • Strict form-level completion remains difficult, so robustness beyond the tested settings is unclear.
Questions for authors
  • How much do results vary across runs and different spreadsheet layouts?
  • Can the semantic judgments be checked with a small independent human evaluation?
3 comments

Dear Reviewer c1ya,

Thank you for your thoughtful review and for recognizing the practical value of CRAFT's grounding, validation, and repair framework. We appreciate your suggestions regarding the reliability of LLM-based semantic evaluation, run-to-run variability, and generalization across spreadsheet layouts. To address these concerns, we have conducted an independent human evaluation of our semantic judge, repeated experiments to assess performance stability, and provided additional analysis across different spreadsheet structures. We hope these results offer a clearer picture of the reliability and limitations of our approach.


[W1 & Q2] Reliability of LLM-Based Semantic Evaluation

First, we would like to clarify that our LLM-based judge is used to assess semantic equivalence between predicted and ground-truth values, instead of subjective response quality. To be specific, it handles cases where semantically equivalent values have different surface forms, such as date formats and naming conventions.

To validate its reliability, we conducted an independent human evaluation. We randomly sampled 200 distinct label–slot pairs requiring LLM-based judgments across both benchmark tracks, covering predictions from CRAFT and the evaluated baselines. Each pair contained non-empty ground-truth and predicted values. Three human annotators independently classified each pair as equivalent, non-equivalent, or uncertain.

The agreement results are summarized below (see Appendix E.2 for the detailed evaluation protocol).

ComparisonMetricResult
Human–HumanFleiss' κ0.747
Human–HumanKrippendorff's α0.747
Human–HumanPairwise Cohen's κ0.660–0.846
Human–LLMAgreement with majority vote90.95% (181/199)
Human–LLMCohen's κ0.803
Human–LLMAgreement on unanimous cases93.94% (155/165)

Among the 18 disagreements with the human majority vote, 13 were judged non-equivalent by the LLM but equivalent by humans, whereas only 5 exhibited the opposite pattern. This suggests a tendency toward more conservative semantic judgments in the sampled cases. One ambiguous instance had no human majority and was excluded from the majority-vote comparison.

Overall, these findings indicate strong agreement between the LLM judge and independent human assessments, while also revealing some ambiguity in semantic equivalence judgments. We have included the detailed evaluation protocol and results in Appendix E.2 of the revised manuscript.


[W2] Limitations and Generalizability

We agree that strict form-level completion remains challenging, as it requires correctly grounding and filling every required slot under potentially irregular spreadsheet layouts. While our additional stability and layout-wise analyses provide further evidence of CRAFT's consistency across the evaluated settings, they do not establish generalization to arbitrary unseen template structures or application domains.

We have clarified the scope of our empirical findings and will further discuss broader generalization as an important direction for future work.


[Q1] Run-to-Run Stability and Layout Variability

To assess run-to-run variability, we conducted three independent runs of CRAFT under identical experimental settings. The results are summarized below (mean ± standard deviation, %, see Table 9 in the revised Appendix E.6).

TrackPair Acc.E2E Acc.
Instruction-Only60.06 ± 0.3817.33 ± 1.83
Multi-File64.86 ± 2.5841.73 ± 1.57

The results show relatively low run-to-run variability, particularly for Pair Accuracy on Instruction-Only. Although Multi-File exhibits greater variability in Pair Accuracy, its E2E Accuracy remains relatively consistent across runs.

We also conducted a layout-wise evaluation across five representative spreadsheet structures, as reported in Appendix E.4 (Fig. 7). CRAFT matches or outperforms the strongest baseline across all five categories, with improvements of 15 percentage points on Mixed-Slot-Layout and 12 points on Repeated-Section.

Together, these results characterize CRAFT's performance variability across repeated executions and different spreadsheet layouts. Although performance varies across settings, the observed improvements are not limited to a single layout category.

Thanks for the response. After reading through the response, I have decided to maintain my original score.

Thank you for taking the time to review our response and for your consideration. We appreciate your feedback.

Reviewer_pvcvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CRAFT, a template-aware agent framework for spreadsheet form filling that consolidates evidence from heterogeneous sources and couples error detection with constrained local repair. It restores corrupted template regions, re-identifies writable slots using a rectangle-aware visual model, and guides refilling while protecting cells judged reliable. For evaluation, the paper also introduces FormFillBench, on which CRAFT outperforms the evaluated baselines.

Critical strengths
  1. The framework addresses an important failure mode in spreadsheet from filling: incorrect edits can corrupt the template and mislead subsequent decisions. Preserving the original template provides a concrete reference for detecting and restoring structural damage.
  2. CRAFT achieves consistent improvements over the evaluated baselines across both benchmark tracks and two backbones. The ablation results further support the contributions of structural validation and slot re-grounding within the full system.
Critical weaknesses
  1. The complexity of the multi-stage framework could be analyzed more thoroughly. Comparing its computational cost with that of the baselines would help assess the performance gains more comprehensively.
  2. While the ablation results support the value of slot re-grounding, the benefit of incorporating it specifically into the diagnosis-and-repair workflow could be more clearly established. A comparison with RASG-guided initial filling would help clarify the value of this design choice.
Questions for authors
  1. In Table 5, several spreadsheet-agent baselines have near-zero fill coverage on the Multi-File track, whereas CRAFT achieves around 90%. Since this metric only measures whether a target field is filled, regardless of value correctness, could the authors provide a more detailed analysis of this gap? For example, what are the common failure modes of these baselines, how does CRAFT address them, and could simple adjustments also substantially improve baseline performance?
7 comments

Dear Reviewer pvcv,

Thank you for your thoughtful review and constructive suggestions. We particularly appreciate your comments on the computational efficiency of CRAFT, the rationale for incorporating RASG into the diagnosis-and-repair workflow, and the low fill coverage observed in spreadsheet-agent baselines. In response, we have analyzed CRAFT's computational costs, conducted additional experiments on initial-stage RASG guidance, and investigated the failure mechanisms underlying low baseline fill coverage. We hope these additional results and analyses help address your concerns and provide a more comprehensive evaluation of our framework.


[W1] Computational Cost and Efficiency

We recognize that CRAFT incurs additional computational overhead due to its multi-stage verification and repair process. Our current work primarily focuses on improving filling reliability, while computational efficiency has not yet been explicitly optimized.

To provide a quantitative assessment, we analyzed CRAFT's LLM token consumption and wall-clock time across both benchmark tracks.

TrackAvg. Tokens / InstanceAvg. Time / Instance (s)
Instruction-Only1,541,032142.78
Multi-File1,256,519132.88

We agree that a direct, standardized cost comparison with all baselines would provide a more comprehensive evaluation of the performance–efficiency trade-off. However, reproducing all baseline experiments to obtain consistent cost statistics would require substantial additional computational resources and time, exceeding our available budget during the rebuttal period. We therefore cannot provide this comparison at present.

We recognize computational efficiency as an important limitation of the current system and a natural next step toward practical deployment. Potential improvements include adaptive verification, cost-aware model routing, and more efficient stopping criteria. The measured computational statistics have been reported in the revised manuscript, as detailed in Section 5.6.


[W2] Effect of Initial RASG Guidance

We thank the reviewer for this insightful and specific suggestion. We conducted an additional experiment in which RASG-generated slot hints are incorporated into initial filling, while the subsequent diagnosis-and-repair workflow remains unchanged.

The results are summarized below, with the complete ablation study provided in Table 3 (Section 5.4) of the revised manuscript.

MethodIO Pair Acc.IO E2E Acc.MF Pair Acc.MF E2E Acc.
CRAFT (Ours)59.6815.5066.5243.31
CRAFT w/ Initial RASG Hints57.4213.5067.4840.94

The results show that adding initial RASG guidance does not consistently improve accuracy. Although it slightly improves pair-level accuracy on the Multi-File track (67.48% vs. 66.52%), it results in lower end-to-end accuracy on both tracks. Moreover, it introduces substantial computational overhead, increasing execution time from approximately 137 s to over 700 s and consuming approximately 0.4M additional tokens. These results support our choice of applying RASG selectively during local repair, rather than introducing additional global grounding during initial filling.

We hypothesize that this behavior is partly related to RASG's recall-oriented design, which is intended for localized suspicious regions. Extending it to the entire form generates more candidate slots and longer hints, potentially introducing redundant information into initial filling. Interestingly, our execution logs show that initial RASG guidance leads to more suspicious-region detections and additional reflection, despite fewer completed repair operations. This suggests that introducing more grounding information early does not necessarily simplify subsequent reasoning.

We also find these observations worth further investigation. In particular, adapting candidate selection thresholds to different grounding scopes may improve the effectiveness of initial-stage guidance. We also plan to investigate the relationship between suspicious-region detection and actual repair effectiveness to better understand how grounding decisions influence downstream correction.

Additionally, we evaluated replacing RASG with direct VLM-based slot grounding, as detailed in our response to Reviewer QK4P.

[Q1] Analysis of Low Fill Coverage in Spreadsheet-Agent Baselines

To investigate the unexpectedly low fill coverage, we conducted a detailed analysis of SheetCopilot's execution logs and output workbooks, identifying three major failure mechanisms.

(1) Incorrect slot grounding. Among 1,323 recorded Write operations, 1,171 (88.5%) targeted non-anchor cells within merged regions, frequently corresponding to labels rather than writable input fields. For example, in one task, the agent attempted to write to B6 within the merged label region A6:C6, whereas the actual input field was D6:H6 (anchored at D6). Consequently, the intended field remained empty. This observation is consistent with the slot-grounding challenges identified in our pilot study and closely relates to the Label–cell overwrite risk signal discussed in our framework.

(2) Insufficient post-write feedback. The tool feedback primarily summarized worksheet dimensions and headers without verifying the contents of target fields. As a result, incorrect writes could remain undetected and uncorrected.

(3) Completion without outcome verification. SheetCopilot's termination mechanism treated the agent's Done response as successful completion without checking whether the target fields have been populated. Consequently, all analyzed tasks were reported as successful at the execution level, despite many target fields remaining empty.

These findings help explain why spreadsheet agents may successfully execute tool calls yet achieve low fill coverage. In contrast, CRAFT explicitly models spreadsheet structure through its dual-view representation and employs reflection and localized repair to identify and correct filling errors. In particular, its Reflective Region Adjudication process explicitly considers risks such as Label–cell overwrite, helping identify modifications that may compromise the original template.

We acknowledge that relatively simple modifications, such as providing agents with explicit writable-slot information, preventing writes to non-empty template cells, or introducing target-field validation after writing, could potentially improve baseline fill coverage. However, these modifications have not been evaluated in our experiments, and their effectiveness remains an open empirical question. While such adjustments may improve fill coverage, their effectiveness remains to be evaluated.

[Q1 Follow-up] Empirical Evaluation of Lightweight Baseline Enhancements

Following our previous analysis of SheetCopilot's three major failure modes, we conducted an additional experiment to examine whether lightweight, targeted modifications could substantially improve baseline performance.

Specifically, we augmented SheetCopilot with three mechanisms corresponding to the identified failure modes:

(1) Pre-filling VLM inspection, which identifies existing content, input regions, and merged-cell anchors to improve slot grounding;

(2) Enhanced post-write feedback, which retrieves actual cell contents and merged-region information to help detect incorrect writes;

(3) VLM-based completion validation, which checks the filled workbook before allowing the agent to terminate and provides feedback for further correction when necessary.

The results are summarized below.

Evaluation TrackMetric (%)Original SheetCopilotEnhanced SheetCopilot
Instruction-OnlyFill Acc42.2242.23
Instruction-OnlyPair Acc33.0232.74
Multi-FileFill Acc1.097.30
Multi-FilePair Acc0.261.22

On the Instruction-Only track, performance remains essentially unchanged. On the Multi-File track, although Fill Acc increases by 6.21 percentage points (from 1.09% to 7.30%), Pair Acc improves by only 0.96 percentage points (from 0.26% to 1.22%).

These results suggest that lightweight enhancements targeting individual failure modes can improve filling coverage but may not yield substantial gains in overall correctness. This further motivates CRAFT's integrated architecture, which systematically combines template understanding, reflective error adjudication, and localized slot re-grounding and repair.

Reviewer_QK4POpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposed CRAFT, a template-aware agent framework for spreadsheet form filling task. It improves the filling accuracy via Reflective Region Adjudication, Rectangle-Aware Slot Grounder (RASG) and constrained local repair. Authors also build a benchmark dataset called FormFillBench which includes 327 forms.

Critical strengths
  1. Authors combine error template restoration, and slot re-grounding and mitigate error propagation via constrained local repair, which I think is valuable.

  2. FormFillBench covers different types of spreadsheet layouts and heterogeneous sources. And CRAFT outperforms 11 baselines, especially in Multi-File Track.

Critical weaknesses
  1. Current ablation only removes the whole PSR, which makes it hard to prove the contribution of RASG and rectangle-aware design. And also, RASG training and FormFillBench all use FUSE and lack the information of template-level data separation.

  2. Pair Accuracy and E2E Accuracy only evals target-entry correctness, instead of directly eval template preservation. Thus, I don't think it can fully prove the key claim of this paper related to template-aware repair.

Questions for authors

How is the training data for RASG isolated from FormFillBench? Could you please describe the data scale and partitioning for each training stage of RASG, specifically the relationship between the FUSE adaptation subset and the FUSE templates found in FormFillBench.

4 comments

Dear Reviewer QK4P,

Thank you for your thoughtful review and for recognizing the value of CRAFT's template-aware repair framework and the diversity of FormFillBench. We particularly appreciate your constructive suggestions regarding the contribution of RASG, template-level data separation, and the evaluation of template preservation. In response, we have conducted additional ablation experiments, clarified our data partitioning procedure, and introduced new template preservation metrics. We hope these additions help address your concerns and strengthen the evaluation of our work.


[W1 & Q1] RASG Ablation and Template-Level Data Isolation

1. Clarification on RASG Training Data and Template-Level Isolation.

We appreciate the reviewer's concern regarding potential data leakage and would like to clarify how RASG's training data were organized. RASG is trained in three stages using synthetic data, CommonForms, and FUSE, with 735/184 (train/validation), 1,272/159/159 (train/validation/test), and 278/40/79 (train/validation/test) samples, respectively. Among these datasets, only FUSE shares its source with a subset of FormFillBench.

To avoid potential data leakage, we identified and held out the benchmark templates before training RASG. Since each FUSE sample corresponds to a distinct form template, all 54 FUSE templates included in FormFillBench belong exclusively to the held-out test split, with no template-level overlap with the training or validation sets. We have added these details to Appendix C.3 of the revised manuscript.

2. Contribution of RASG and Rectangle-Aware Grounding.

To better isolate the contribution of RASG from the overall PSR module, we conducted an additional ablation study in which RASG is replaced with direct VLM-based slot grounding, while keeping the remaining diagnosis-and-repair workflow unchanged. The results are summarized below, with the complete ablation study provided in Table 3 (Section 5.4) of the revised manuscript.

MethodIO Pair Acc.IO E2E Acc.MF Pair Acc.MF E2E Acc.
CRAFT59.6815.5066.5243.31
CRAFT w/ VLM Grounding55.3213.5030.4611.81

This replacement reduces Pair Accuracy from 59.68% to 55.32% on Instruction-Only track and from 66.52% to 30.46% on Multi-File track. Notably, Multi-File E2E Accuracy drops from 43.31% to 11.81%.

These results provide more direct evidence for the contribution of RASG's specialized rectangle-aware grounding, beyond the benefits of the overall diagnosis-and-repair workflow.

[W2] Template Preservation Evaluation

We sincerely thank the reviewer for this valuable observation. We agree that preserving the original template is an important requirement in practical spreadsheet form filling, and that our existing Pair Accuracy and E2E Accuracy metrics do not directly evaluate unintended template modifications.

To be specific, our original evaluation adopts a structured data collection perspective, where each spreadsheet form could contain a predefined set of target slots, similar to a structured questionnaire (like Microsoft Forms). Under this formulation, successful filling is primarily determined by whether the correct values are written to their designated locations, which motivates our use of Pair Accuracy and E2E Accuracy. Accordingly, the primary motivation for template awareness in CRAFT is to maintain reliable label–slot relationships throughout the filling process, preventing template corruption from misleading subsequent grounding and filling decisions.

However, as the reviewer insightfully points out, a completed spreadsheet is also a document in its own right. Beyond collecting correct values, practical applications may require the original template to remain intact and usable. We fully agree that this is an important and complementary evaluation perspective, which was not explicitly captured by our original metrics.

Motivated by this observation, we conducted an additional template preservation evaluation across both benchmark tracks. Specifically, we introduced two complementary metrics: Macro Preservation, which averages cell-level preservation rates across forms, and E2E Preservation, which requires all protected template cells within a form to remain unchanged. The results are summarized below and reported in Table 2 (Section 5.3), with additional details in Table 7 and Appendix E.3 of the revised manuscript.

MethodIO MacroIO E2E Pre.MF MacroMF E2E Pre.
CRAFT94.9341.0099.4992.91
VLM + HTML + Image94.7339.0092.6249.61
OpenAI Agents SDK90.1025.5089.8865.35
SheetCopilot89.3623.5084.020.00

CRAFT achieves 94.93% / 99.49% macro-level preservation and 41.00% / 92.91% form-level preservation on Instruction-Only and Multi-File tracks, respectively, while maintaining the highest Pair Accuracy on both tracks. We have incorporated these results into the revised manuscript to provide a more comprehensive evaluation of reliable spreadsheet form filling, as detailed in Section 5.3 and Appendix E.3.

Thanks for authors' clarification and experiments. These addressed my concerns and hope to see these in final paper. I keep my positive score.

Thank you for your encouraging feedback and thoughtful consideration. We are glad that our clarifications and additional experiments have addressed your concerns. We will make sure to incorporate these improvements into the final version of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.